블로그 이미지
윤영식
Full Stacker, Application Architecter, KnowHow Dispenser and Bike Rider

Publication

Category

Recent Post

2013. 9. 13. 21:19 MongoDB/Prototyping

MongoDB에 구글의 도서검색 내역을 넣고, 여기서 도서의 Description을 하둡으로 분석하여 추천도서를 만들어 보자 



구글도서에서 Description을 MongoDB에 저장하기 

  - 이클립스에서 Maven Project를 하나 생성하고, pom.xml 을 다음과 같이 구성한다 

    자바에서 몽고디비를 사용하기 위한 드라이버와 구글 검색결과(JSON)을 파싱하기위한 JSON라이브러리를 추가한다 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.mobiconsoft</groupId>

  <artifactId>booksearch</artifactId>

  <version>0.0.1-SNAPSHOT</version>

  <dependencies>

  <dependency>

  <groupId>org.mongodb</groupId>

  <artifactId>mongo-java-driver</artifactId>

  <version>2.11.2</version>

  </dependency>


  <dependency>

  <groupId>junit</groupId>

  <artifactId>junit</artifactId>

  <version>4.10</version>

  </dependency>

 

  <dependency>

  <groupId>org.json</groupId>

  <artifactId>json</artifactId>

  <version>20090211</version>

  </dependency>

  </dependencies>

</project>

  - BookSearcher.java 코딩 (편의상 import 구문 제외)

public class BookSearcher {


  // 도서검색 

  public String searchBooks(String keyword) {

    URL url = null;

    try {

      url = new URL("https://www.googleapis.com/books/v1/volumes?q=" + keyword);

    } catch (MalformedURLException e) {

      e.printStackTrace();

    }

    

    StringBuffer sb = new StringBuffer();

    String line;

    try {

      URLConnection urlConn = url.openConnection();

      BufferedReader br = new BufferedReader(new InputStreamReader(urlConn.getInputStream(), "utf-8"));

      while((line = br.readLine()) != null) sb.append(line);

    } catch(IOException e) {

      e.printStackTrace();

    }

    

    return sb.toString();

  }

  

  // 도서 검색 결과에서  items찾아와 저장한다 

  public void saveBooks(String books) {

    Mongo mongo = null;

    try{

      mongo = new MongoClient("localhost", 27017);

    } catch(Exception e) {

      e.printStackTrace();

      throw new RuntimeException();

    }

    

   // 몽고디비 db는 books-db 이고 컬렉션은 books 로 만들어짐 

    mongo.setWriteConcern(new WriteConcern(1, 2000));

    DB bookDB = mongo.getDB("books-db");

    DBCollection bookColl = bookDB.getCollection("books");

    

    try{

      JSONObject json = new JSONObject(books);

      JSONArray items = json.getJSONArray("items");

      for( int i=0; i<items.length(); i++) {

        DBObject doc = new BasicDBObject();

        // search-book key로 value가 들어간다

        doc.put("search-book", (DBObject)JSON.parse(items.getJSONObject(i).toString()));

        bookColl.save(doc);

      }

    } catch(JSONException e) {

      e.printStackTrace();

    }

  }

}

  - 테스트 해보자 

    JUnit 테스트전에 mongodb를 기동한다 

// 몽고디비 

$ ../bin/mongod -dbpath=/Users/dowon/Documents/mongodb/database


// 테스트 

public class BookSearcherTest {

  

    private BookSearcher bookSearcher;

    

    @Before

    public void setUp() throws Exception {

      this.bookSearcher = new BookSearcher();

    }


    // search 결과 보기 

    @Test

    public void testSearchBooks() throws Exception {

      String result = this.bookSearcher.searchBooks("nosql");

      //assertNotNull(result);

      System.out.println(result);

    }

    

    // 데이터 저장하기 

    @Test

    public void testSaveBooks() throws Exception {

      String result = this.bookSearcher.searchBooks("nosql");

      this.bookSearcher.saveBooks(result);

    }

}


// 테스트 성공후 mongo 쉘을 통하여 확인 

> use books-db

switched to db books-db

> show collections

books

system.indexes

> db.books.find().length();

10

> db.books.find()

{ "_id" : ObjectId("5232e69cda06561b2e11306c"), "search-book" : { "saleInfo" : { "saleability" : "NOT_FOR_SALE", "isEbook" : false, "country" : "KR" }, "id" : "tv5iO9MnObUC", "searchInfo" : { "textSnippet" : "They provide examples, practical solutions, and expert education in new technologies, all designed to help programmers do a better job. wrox.com Programmer Forums Join our Programmer to Programmer forums to ask and answer programming ..." }, "etag" : "HX8hesQgrJM", "volumeInfo" : { "pageCount" : 408, "averageRating" : 3, "infoLink" : "http://books.google.co.kr/books?id=tv5iO9MnObUC&dq=nosql&hl=&source=gbs_api", "printType" : "BOOK", "publisher" : "John Wiley & Sons", "authors" : [  "Shashank Tiwari" ], "canonicalVolumeLink" : "http://books.google.co.kr/books/about/Professional_NoSQL.html?hl=&id=tv5iO9MnObUC", "title" : "Professional NoSQL", "previewLink ... 중략 ...



MongoDB Hadoop Connector 사용하기  

  - 몽고디비와 하둡을 연결하는 방법을 제공한다 

     https://github.com/mongodb/mongo-hadoop 에서 1.1.x 의 Core 다운로드 한다 (mongo-hadoop-core_1.1.2-1.1.0.jar)

  - Input-Output으로 몽고디비를 사용할 경우   

    

 - 분석을 위하여 Pig, MR을 할 경우

    

  - ETL처럼 처리후 별도의 저장소로 던져질 경우

    ETL from MongoDB

      

    ETL to MongoDB

        


  - Eclipse에 새로운 Book Search Mapper와 Reducer 프로젝트를 만들고 pom.xml 을 만든다 

    mongo-hadoop-core  파일을 maven에 등록되어 있지 않기때문에 수동으로 .m2/repository에 만들어 주어야 한다 

    예)

    > 레파지토리 : /Users/dowon/.m2/repository

    > 파일위치 : mongo-hadoop-core/mongo-hadoop-core_1.1.2/1.1.0/mongo-hadoop-core_1.1.2-1.1.0.jar

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.mobiconsoft</groupId>

  <artifactId>booksearch_mapreduce</artifactId>

  <version>0.0.1-SNAPSHOT</version>

  <dependencies>

    <dependency>

      <groupId>org.apache.hadoop</groupId>

      <artifactId>hadoop-core</artifactId>

      <version>1.1.2</version>

    </dependency>

    

    <dependency>

      <groupId>org.mongodb</groupId>

      <artifactId>mongo-java-driver</artifactId>

      <version>2.11.2</version>

    </dependency>

    

    <!-- 수동 설정 --> 

    <dependency>

      <groupId>mongo-hadoop-core</groupId>

      <artifactId>mongo-hadoop-core_1.1.2</artifactId>

      <version>1.1.0</version>

    </dependency>

  </dependencies>

  

  <build>

    <plugins>

      <plugin>

        <artifactId>maven-antrun-plugin</artifactId>

        <configuration>

          <tasks>

            <copy file="target/${project.artifactId}-${project.version}.jar"

              tofile="/Users/dowon/Documents/input/${project.artifactId}-${project.version}.jar" />

          </tasks>

        </configuration>

        <executions>

          <execution>

            <phase>install</phase>

            <goals>

              <goal>run</goal>

            </goals>

          </execution>

        </executions>

      </plugin>

    </plugins>

  </build>

  

</project>

  - Mapper와 Reducer 클래스를 코딩 

// Mapper

public class BookSearchMapper extends Mapper<Object, BSONObject, Text, IntWritable> {

  

  private final static IntWritable ONE = new IntWritable();

  private Text word = new Text();

  

  protected void map(Object key, BSONObject value, Context context) 

    throws IOException, InterruptedException {

    BasicDBObject anItem = (BasicDBObject)value.get("search-book");

    BasicDBObject volumeInfo = (BasicDBObject)anItem.get("volumeInfo");

    String description = volumeInfo.getString("description");

    if(description == null || description.trim().length() <= 0) return;

    

    StringTokenizer st = new StringTokenizer(description);

    while(st.hasMoreTokens()) {

      word.set(st.nextToken());

      context.write(word, ONE);

    }

  }

}


// Reducer

public class BookSearcherReducer extends

  Reducer<Text, IntWritable, Text, IntWritable> {

  

  protected void reduce(Text key, Iterable<IntWritable> values, Context context) 

    throws IOException, InterruptedException {

    int sum = 0;

    for(final IntWritable value : values) sum += value.get();

    context.write(key, new IntWritable(sum));

  }

}

  - Job을 만든다

public class MongoJob extends MongoTool {

  static {

    Configuration.addDefaultResource("mongo-default.xml");

    Configuration.addDefaultResource("mongo-book.xml");

  }

  

  public static void main(String[] args) throws Exception {

    Configuration conf = new Configuration();

    

    JobHelper.addJarForJob(conf, "/Users/dowon/.m2/repository/mongo-hadoop-core/mongo-hadoop-core_1.1.2/1.1.0/mongo-hadoop-core_1.1.2-1.1.0.jar:"

          + "/Users/dowon/.m2/repository/org/mongodb/mongo-java-driver/2.11.2/mongo-java-driver-2.11.2.jar");

    

    System.exit(ToolRunner.run(conf, new MongoJob(), args));

  }

}

  - 리소스 xml을 만든다 

    https://github.com/mongodb/mongo-hadoop/blob/master/examples/treasury_yield/src/main/resources/mongo-defaults.xml 

   에서 xml 정보를 copy 하여 mongo-book.xml 을 만든 후 하기 내용을 수정하여 입력해야 한다

<property>

    <!-- Class for the mapper -->

    <name>mongo.job.mapper</name>

    <value>booksearch_mapreduce.BookSearchMapper</value>

  </property>

  <property>

    <!-- Reducer class -->

    <name>mongo.job.reducer</name>

    <value>booksearch_mapreduce.BookSearcherReducer</value>

  </property>

  <property>

    <!-- InputFormat Class -->

    <name>mongo.job.input.format</name>

    <value>com.mongodb.hadoop.MongoInputFormat</value>

  </property>

  <property>

    <!-- OutputFormat Class -->

    <name>mongo.job.output.format</name>

    <value>com.mongodb.hadoop.MongoOutputFormat</value>

  </property>

  <property>

    <!-- Output key class for the output format -->

    <name>mongo.job.output.key</name>

    <value>org.apache.hadoop.io.Text</value>

  </property>

  <property>

    <!-- Output value class for the output format -->

    <name>mongo.job.output.value</name>

    <value>com.mongodb.hadoop.io.BSONWritable</value>

  </property>

  <property>

    <!-- Output key class for the mapper [optional] -->

    <name>mongo.job.mapper.output.key</name>

    <value>org.apache.hadoop.io.Text</value>

  </property>

  <property>

    <!-- Output value class for the mapper [optional] -->

    <name>mongo.job.mapper.output.value</name>

    <value>org.apache.hadoop.io.IntWritable</value>

  </property>

  <property>

    <!-- Class for the combiner [optional] -->

    <name>mongo.job.combiner</name>

    <value>booksearch_mapreduce.BookSearcherReducer</value>

  </property>

  - "Mave Build..." clean install 하여 .jar  파일을 만든다 (참조에 첨부파일)

 - 다음 하둡 runtime(start-all.sh) 을 수행한다 

  - 하둡 수행 쉘을 만든다 

//////////////////////

// mongodb.sh 내역 

#!/bin/sh


export REPO=/Users/dowon/.m2/repository

export MONGO_DRIVER=$REPO/org/mongodb/mongo-java-driver/2.11.2/mongo-java-driver-2.11.2.jar

export MONGO_HADOOP=$REPO/mongo-hadoop-core/mongo-hadoop-core_1.1.2/1.1.0/mongo-hadoop-core_1.1.2-1.1.0.jar

export HADOOP_CLASSPATH=$MONGO_DRIVER:$MONGO_HADOOP

export HADOOP_USER_CLASSPATH_FIRST=true


hadoop jar booksearch_mapreduce-0.0.1-SNAPSHOT.jar booksearch_mapreduce.MongoJob



/////////////

/// 수행하기 

$ mongodb.sh

2013-09-13 20:53:24.630 java[1431:1203] Unable to load realm info from SCDynamicStore

13/09/13 20:53:24 INFO util.MongoTool: Created a conf: 'Configuration: core-default.xml, core-site.xml, mongo-default.xml, mongo-book.xml, mapred-default.xml, mapred-site.xml' on {class booksearch_mapreduce.MongoJob} as job named '<unnamed MongoTool job>'

13/09/13 20:53:24 INFO util.MongoTool: Mapper Class: class booksearch_mapreduce.BookSearchMapper

13/09/13 20:53:24 INFO util.MongoTool: Setting up and running MapReduce job in foreground, will wait for results.  {Verbose? false}

13/09/13 20:53:25 INFO util.MongoSplitter: MongoSplitter calculating splits

13/09/13 20:53:25 INFO util.MongoSplitter: use range queries: false

.. 중략 ...

13/09/13 20:53:41 INFO mapred.JobClient:     Spilled Records=1428

13/09/13 20:53:41 INFO mapred.JobClient:     Map output bytes=14049

13/09/13 20:53:41 INFO mapred.JobClient:     Total committed heap usage (bytes)=269619200

13/09/13 20:53:41 INFO mapred.JobClient:     Combine input records=1299

13/09/13 20:53:41 INFO mapred.JobClient:     SPLIT_RAW_BYTES=195

13/09/13 20:53:41 INFO mapred.JobClient:     Reduce input records=714

13/09/13 20:53:41 INFO mapred.JobClient:     Reduce input groups=714

13/09/13 20:53:41 INFO mapred.JobClient:     Combine output records=714

13/09/13 20:53:41 INFO mapred.JobClient:     Reduce output records=714

13/09/13 20:53:41 INFO mapred.JobClient:     Map output records=1299



MongoDB에서 결과값 확인하기 

  - 브라우져에서 결과값을 확인하고 싶다면 몽고디비에 옵션으로 --rest 를 주면 28017 포트로 RESTful 하게 호출할 수 있다 

 $ ./bin/mongod -dbpath=/Users/dowon/Documents/mongodb/database --rest 

  - 결과 화면 

    결과값은 out 컬렉션에 생성이 된다


<참조>

  - 검색 책 정보 

books.json


  - 이클립스 project workspace

booksearch.tar


  - 하둡기동후 수행하는 쉘 

mongodb.sh


  - 반출한 booksearch mapreducer jar 파일 

booksearch_mapreduce-0.0.1-SNAPSHOT.jar






posted by 윤영식
2013. 7. 27. 16:06 MongoDB/Prototyping

몽고디비에서 Shard 1개 ( = ReplicaSet 1개)를 구성하였다면 이제는 n개의 Shard를 구성하여 mongos를 통하여 연결해 본다




1. Shard n개 구성하기

  - Shard Server 1개에 mongod 1개만 만듦 (mongod n개 구성은 다음 블로깅에서)

  - db11, db22, db33 디렉토리를 미리 만들어 놓는다

/mongodb_2.4.5> mongod --shardsvr --dbpath /mongodb_2.4.5/db11 --port 10000
Sat Jul 27 15:15:40.854 [initandlisten] MongoDB starting : pid=71105 port=10000 dbpath=/mongodb_2.4.5/db11 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 15:15:41.141 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 271ms
Sat Jul 27 15:15:41.141 [websvr] admin web console waiting for connections on port 11000
Sat Jul 27 15:15:41.142 [initandlisten] waiting for connections on port 10000


/mongodb_2.4.5> mongod --shardsvr --dbpath /mongodb_2.4.5/db22 --port 20000
Sat Jul 27 15:16:38.842 [initandlisten] MongoDB starting : pid=71106 port=20000 dbpath=/mongodb_2.4.5/db22 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 15:16:39.105 [websvr] admin web console waiting for connections on port 21000
Sat Jul 27 15:16:39.105 [initandlisten] waiting for connections on port 20000


/mongodb_2.4.5> mongod --shardsvr --dbpath /mongodb_2.4.5/db33 --port 30000
Sat Jul 27 15:18:04.778 [initandlisten] MongoDB starting : pid=71120 port=30000 dbpath=/mongodb_2.4.5/db33 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 15:18:05.031 [websvr] admin web console waiting for connections on port 31000
Sat Jul 27 15:18:05.031 [initandlisten] waiting for connections on port 30000



2. Config 환경 구성하기

   - config1, config2, config3 디렉토리를 미리 만들어 놓는다

/mongodb_2.4.5> mongod --help
.. 중략 ..
Master/slave options (old; use replica sets instead):
  --master              master mode
  --slave               slave mode
  --source arg          when slave: specify master as <server:port>
  --only arg            when slave: specify a single database to replicate
  --slavedelay arg      specify delay (in seconds) to be used when applying
                        master ops to slave
  --autoresync          automatically resync if slave data is stale

Replica set options:
  --replSet arg           arg is <setname>[/<optionalseedhostlist>]
  --replIndexPrefetch arg specify index prefetching behavior (if secondary)
                          [none|_id_only|all]

Sharding options:
  --configsvr           declare this is a config db of a cluster; default port
                        27019; default dir /data/configdb
  --shardsvr            declare this is a shard db of a cluster; default port
                        27018


  - config 서버 기동하기

/mongodb_2.4.5> mongod --configsvr --dbpath /mongodb_2.4.5/config1 --port 40000
Sat Jul 27 15:22:12.404 [initandlisten] MongoDB starting : pid=71141 port=40000 dbpath=/mongodb_2.4.5/config1 master=1 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 15:22:12.544 [initandlisten] creating replication oplog of size: 5MB...
Sat Jul 27 15:22:12.547 [initandlisten] ******
Sat Jul 27 15:22:12.547 [websvr] admin web console waiting for connections on port 41000
Sat Jul 27 15:22:12.547 [initandlisten] waiting for connections on port 40000



3. Mongos로 Shard 제어하기

  - 명령어 : mongos <config server>

/mongodb_2.4.5> mongos --help
.. 중략 ..
Sharding options:
  --configdb arg      1 or 3 comma separated config servers
  --localThreshold arg 

                              ping time (in ms) for a node to be considered local (default 15ms)
  --test                   just run unit tests
  --upgrade             upgrade meta data version
  --chunkSize arg     maximum amount of data per chunk (기본 : 64kbytes)
  --ipv6                   enable IPv6 support (disabled by default)
  --jsonp                 allow JSONP access via http (has security implications)
  --noscripting          disable scripting engine


  - mongos 연결하기

/mongodb_2.4.5> mongos --configdb localhost:40000 --chunkSize 1 --port 50000
Sat Jul 27 15:31:45.926 [mongosMain] options: { chunkSize: 1, configdb: "localhost:40000", port: 50000 }
Sat Jul 27 15:31:45.929 [Balancer] about to contact config servers and shards
Sat Jul 27 15:31:45.929 [websvr] admin web console waiting for connections on port 51000
Sat Jul 27 15:31:45.929 [Balancer] config servers and shards contacted successfully
Sat Jul 27 15:31:45.929 [mongosMain] waiting for connections on port 50000
Sat Jul 27 15:31:45.932 [Balancer] distributed lock 'balancer/nulpulum-mac-13-retina.local:50000:1374906705:16807' acquired, ts : 51f36951133d3c7f4fe7b076
Sat Jul 27 15:31:45.932 [Balancer] distributed lock 'balancer/nulpulum-mac-13-retina.local:50000:1374906705:16807' unlocked.



4. REPL로 데이터 제어하

  - mongo로 들어가면 mongos 프롬프트가 나온다

mongodb_2.4.5> mongo localhost:50000
MongoDB shell version: 2.4.5
connecting to: localhost:50000/test
mongos>

mongos> show dbs
admin    (empty)
config    0.046875GB


  - admin db로 들어가서 config server에 shard환경을 구성한다

mongos> show dbs
admin    (empty)
config    0.046875GB
mongos> use admin
switched to db admin
mongos> show collections
mongos> db
admin
mongos> db.runCommand({ addShard: 'localhost:10000' });
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> db.runCommand({ addShard: 'localhost:20000' });
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> db.runCommand({ addShard: 'localhost:30000' });
{ "shardAdded" : "shard0002", "ok" : 1 }


  - 동적으로 mongod를 Shard에 포함시키기

/////////////// --shardsvr 옵션없이 mongod를 띄운다

/mongodb_2.4.5> mongod --dbpath /mongodb_2.4.5/db111 --port 60000
Sat Jul 27 15:43:48.510 [initandlisten] MongoDB starting : pid=71193 port=60000 dbpath=/mongodb_2.4.5/db111 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 15:43:48.542 [websvr] admin web console waiting for connections on port 61000
Sat Jul 27 15:43:48.542 [initandlisten] waiting for connections on port 60000
Sat Jul 27 15:43:57.418 [initandlisten] connection accepted from 127.0.0.1:51723 #1 (1 connection now open)


/////////////// runCommand를 수행한다

mongos> show collections
mongos> db.runCommand({ addShard: 'localhost:60000' });
{ "shardAdded" : "shard0003", "ok" : 1 }


/////////////// mongos 의 system out 출력값
Sat Jul 27 15:43:57.419 [conn1] going to add shard: { _id: "shard0003", host: "localhost:60000" }



5. 데이터 Sharding 하기

  - mongos를 통하여 데이터를 Sharding하여 나누어서 넣는 방법 : 한곳으로 데이터가 들어갈때 트래픽이 감당하기 힘들때 다른 샤드로 데이터를 보낸다 (10gen)

mongos> use dowonDB
switched to db dowonDB
mongos> db
dowonDB
mongos> db.person.save({age:1, name:'youngsik', address:'seoul'});
mongos> db.person.find();
{ "_id" : ObjectId("51f36cfb065189f0c02620d4"), "age" : 1, "name" : "youngsik", "address" : "seoul" }
mongos> use admin
switched to db admin
mongos> db
admin
mongos> db.runCommand({ enablesharding: 'dowonDB' })
{ "ok" : 1 }
mongos> db.runCommand({ shardcollection: 'dowonDB.person', key:{_id:1} })
{ "collectionsharded" : "dowonDB.person", "ok" : 1 }


  - 데이터 넣어서 샤드 점검하기 : 여러 mongod 서버로 데이터가 나뉘어서 들어갈 것이다. 즉, mongos가 로드발샌싱하여 데이터를 라우팅하여 줄 것이다. (5백만건 넣기)

mongos> use dowonDB
switched to db dowonDB
mongos> db
dowonDB
mongos> for(var i=0; i < 5000000 ; i++) { db.person.save({ age: i, name:i+9, address: i+1000}); };

mongos> db.person.find().size();
5000001

mongos> db.person.find().skip(1000000).limit(10);
{ "_id" : ObjectId("51f36fc4065189f0c037d1d1"), "age" : 1159420, "name" : 1159429, "address" : 1160420 }
{ "_id" : ObjectId("51f36f8d065189f0c02b7e4c"), "age" : 351607, "name" : 351616, "address" : 352607 }
{ "_id" : ObjectId("51f37031065189f0c04dc534"), "age" : 2597983, "name" : 2597992, "address" : 2598983 }
{ "_id" : ObjectId("51f36fab065189f0c032416e"), "age" : 794777, "name" : 794786, "address" : 795777 }
{ "_id" : ObjectId("51f36fc4065189f0c037d1d2"), "age" : 1159421, "name" : 1159430, "address" : 1160421 }
{ "_id" : ObjectId("51f36f8d065189f0c02b7e4d"), "age" : 351608, "name" : 351617, "address" : 352608 }
{ "_id" : ObjectId("51f37031065189f0c04dc535"), "age" : 2597984, "name" : 2597993, "address" : 2598984 }
{ "_id" : ObjectId("51f36fab065189f0c032416f"), "age" : 794778, "name" : 794787, "address" : 795778 }
{ "_id" : ObjectId("51f36fc4065189f0c037d1d3"), "age" : 1159422, "name" : 1159431, "address" : 1160422 }
{ "_id" : ObjectId("51f36f8d065189f0c02b7e4e"), "age" : 351609, "name" : 351618, "address" : 352609 }


  - db11, db22, db33, db111 각 디렉토리에 1Gbytes 가량의 데이터가 쌓여진다.



<참조>

  - 10Gen ReplicaSet 프리젠테이션

posted by 윤영식
2013. 7. 27. 14:56 MongoDB/Prototyping

몽고디비에서 Replica Set 환경을 구성해 본다. ReplicaSet을 하나의 Shard로 만들어서 여러개의 Shard가 구성되면 하기와 같이 routing은 mongos가 해준다




1. Replica Set 만들기

  - 옵션 : --replSet 

  - 디렉토리 만들어 놓기 : db1, db2, db3을 미리 만들어 놓는다

mongodb_2.4.5> mongod --replSet dowonSet --port  20000  -dbpath /mongodb_2.4.5/db1
Sat Jul 27 14:11:58.007 [initandlisten] MongoDB starting : pid=71010 port=20000 dbpath=/mongodb_2.4.5/db1 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 14:11:58.299 [websvr] admin web console waiting for connections on port 21000
Sat Jul 27 14:11:58.300 [initandlisten] waiting for connections on port 20000
Sat Jul 27 14:12:08.302 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)


mongodb_2.4.5> mongod --replSet dowonSet --port  30000  -dbpath /mongodb_2.4.5/db2
Sat Jul 27 14:32:08.934 [initandlisten] MongoDB starting : pid=71062 port=30000 dbpath=/mongodb_2.4.5/db2 64-bit host=nulpulum-mac-13-retina.local


/mongodb_2.4.5> mongod --replSet dowonSet --port  40000  -dbpath /mongodb_2.4.5/db3
Sat Jul 27 14:32:16.317 [initandlisten] MongoDB starting : pid=71063 port=40000 dbpath=/mongodb_2.4.5/db3 64-bit host=nulpulum-mac-13-retina.local



2. Replica Set 환경설정하기

  - mongo REPL 로 연결

  - config 환경 구성

/mongodb_2.4.5> mongo localhost:20000
MongoDB shell version: 2.4.5
connecting to: localhost:20000/test
> var config={_id:'dowonSet', members:[{_id:0, host:'localhost:20000'}, {_id:1, host:'localhost:30000'}, {_id:2, host:'localhost:40000'}] };


///////////// Replica set help 보기

> rs.help();
    rs.status()                         { replSetGetStatus : 1 } checks repl set status
    rs.initiate()                         { replSetInitiate : null } initiates set with default settings
    rs.initiate(cfg)                   { replSetInitiate : cfg } initiates set with configuration cfg
    rs.conf()                           get the current configuration object from local.system.replset
    rs.reconfig(cfg)                  updates the configuration of a running replica set with cfg (disconnects)
    rs.add(hostportstr)             add a new member to the set with default attributes (disconnects)
    rs.add(membercfgobj)        add a new member to the set with extra attributes (disconnects)
    rs.addArb(hostportstr)        add a new member which is arbiterOnly:true (disconnects)
    rs.stepDown([secs])          step down as primary (momentarily) (disconnects)
    rs.syncFrom(hostportstr)    make a secondary to sync from the given member
    rs.freeze(secs)                  make a node ineligible to become primary for the time specified
    rs.remove(hostportstr)       remove a host from the replica set (disconnects)
    rs.slaveOk()                      shorthand for db.getMongo().setSlaveOk()

    db.isMaster()                     check who is primary
    db.printReplicationInfo()       check oplog size and time range

    reconfiguration helpers disconnect from the database so the shell will display
    an error, even if the command succeeds.
    see also http://<mongod_host>:28017/_replSet for additional diagnostic info


///////////// 환경 적용하기

> rs.initiate(config);
{
    "info" : "Config now saved locally.  Should come online in about a minute.",
    "ok" : 1
}


///////////// Enter를 치면 프롬프트가 변경되어 나온다

dowonSet:PRIMARY>
dowonSet:PRIMARY> show dbs
local    0.328125GB
dowonSet:PRIMARY> use dowonDB
switched to db dowonDB
dowonSet:PRIMARY> db
dowonDB
dowonSet:PRIMARY>



3. REPL 조작하기

  - PRIMARY에서 데이터 조작하기

dowonSet:PRIMARY> db.person.save({age:23, name:'youngsik', title:'hi'});
dowonSet:PRIMARY> show collections
person
system.indexes
dowonSet:PRIMARY> db.person.find();
{ "_id" : ObjectId("51f35c76076489dd83ccb46d"), "age" : 23, "name" : "youngsik", "title" : "hi" }


  - SECONDARY 들어가 보기 : PRIMARY가 다운되면 show collections에서 에러가 발생하지 않고 자신이 PRIMARY로 된다

    PRIMARY 1개 + SECONDARY 2개 + 여분 n개

/mongodb_2.4.5> mongo localhost:30000
MongoDB shell version: 2.4.5
connecting to: localhost:30000/test
dowonSet:SECONDARY> show dbs
dowonDB    0.203125GB
local    0.328125GB
test    (empty)
dowonSet:SECONDARY> use dowonDB
switched to db dowonDB
dowonSet:SECONDARY> db
dowonDB
dowonSet:SECONDARY> show collections
Sat Jul 27 14:42:38.244 JavaScript execution failed: error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:L128

///////////// 20000 번 포트의 mongod를 강제로 다운시켰을 때 : 40000 포트의 mongod가 PRIMARY가 된다

/mongodb_2.4.5> mongo localhost:40000
MongoDB shell version: 2.4.5
connecting to: localhost:40000/test
dowonSet:PRIMARY>
dowonSet:PRIMARY> use dowonDB
switched to db dowonDB
dowonSet:PRIMARY> show collections
person
system.indexes
dowonSet:PRIMARY> db.person.find();
{ "_id" : ObjectId("51f35c76076489dd83ccb46d"), "age" : 23, "name" : "youngsik", "title" : "hi" }


  - SECONDARY로 들어가서 show collections가 안될 경우 rs.slaveOk() 명령수행

/mongodb_2.4.5> mongo localhost:30000
MongoDB shell version: 2.4.5
connecting to: localhost:30000/test
dowonSet:SECONDARY>
dowonSet:SECONDARY> show dbs
dowonDB    0.203125GB
local    0.328125GB
test    (empty)
dowonSet:SECONDARY> use dowonDB
switched to db dowonDB
dowonSet:SECONDARY> db
dowonDB
dowonSet:SECONDARY> show collections
Sat Jul 27 14:48:38.663 JavaScript execution failed: error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:L128
dowonSet:SECONDARY> rs.slaveOk();
dowonSet:SECONDARY> show collections
person
system.indexes
dowonSet:SECONDARY> db.person.find();
{ "_id" : ObjectId("51f35c76076489dd83ccb46d"), "age" : 23, "name" : "youngsik", "title" : "hi" }


현재까지 Shard 1개를 만들었다. (구현되지 않는 것은 거짓말이다)



<참조>

  - 10gen Sharding 메뉴얼

posted by 윤영식
2013. 7. 27. 12:28 MongoDB/Prototyping

몽고디비의 마스터/슬레이브 환경을 만들어서 fault tolerance 시스템을 구성해 본다



1. 마스터 만들기

  - 옵션 : --master

mongodb_2.4.5> mongod --dbpath /mongodb_2.4.5/master --master --port 38000
Sat Jul 27 11:50:39.279 [initandlisten] MongoDB starting : pid=70914 port=38000 dbpath=mongodb_2.4.5/master master=1 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 11:50:39.279 [initandlisten] db version v2.4.5
.. 중략 ..
Sat Jul 27 11:50:39.347 [websvr] admin web console waiting for connections on port 39000
Sat Jul 27 11:50:39.347 [initandlisten] waiting for connections on port 38000
Sat Jul 27 11:51:47.344 [initandlisten] connection accepted from 127.0.0.1:63461 #1 (1 connection now open)


mongodb_2.4.5> mongo --port 38000
MongoDB shell version: 2.4.5
connecting to: 127.0.0.1:38000/test
> show dbs
local    0.328125GB

> db.printReplicationInfo()
configured oplog size:   192MB
log length start to end: 317secs (0.09hrs)
oplog first event time:  Sat Jul 27 2013 11:50:16 GMT+0900 (KST)
oplog last event time:   Sat Jul 27 2013 11:55:33 GMT+0900 (KST)
now:                     Sat Jul 27 2013 11:55:36 GMT+0900 (KST)

> db.printSlaveReplicationInfo()
local.sources is empty; is this db a --slave?
> db.printShardingStatus()
printShardingStatus: this db does not have sharding enabled. be sure you are connecting to a mongos from the shell and not to a mongod.



2. slave 만들기

  - 옵션 : --slave  --source <master 위치>

mongodb_2.4.5> mongod --dbpath /mongodb_2.4.5/slave --slave --port 48000 --source localhost:38000
Sat Jul 27 12:04:55.853 [initandlisten] MongoDB starting : pid=70927 port=48000 dbpath=mongodb_2.4.5/slave slave=1 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 12:04:55.853 [initandlisten] options: { dbpath: "/mongodb_2.4.5/slave", port: 48000, slave: true, source: "localhost:38000" }
Sat Jul 27 12:04:56.891 [replslave] build index done.  scanned 0 total records. 0 secs


  - master의 내역확인: slave하나가 연결되었다

Sat Jul 27 11:51:47.344 [initandlisten] connection accepted from 127.0.0.1:63461 #1 (1 connection now open)
Sat Jul 27 12:04:56.889 [initandlisten] connection accepted from 127.0.0.1:63547 #2 (2 connections now open)
Sat Jul 27 12:04:57.896 [slaveTracking] build index local.slaves { _id: 1 }
Sat Jul 27 12:04:57.897 [slaveTracking] build index done.  scanned 0 total records. 0 secs


  - slave 하나더 만들기

mongodb_2.4.5> mongod --dbpath /mongodb_2.4.5/slave2 --slave --port 58000 --source localhost:38000
Sat Jul 27 12:08:53.817 [initandlisten] MongoDB starting : pid=70932 port=58000 dbpath=mongodb_2.4.5/slave2 slave=1 64-bit host=nulpulum-mac-13-retina.local
Sat Jul 27 12:08:53.818 [initandlisten] options: { dbpath: "/mongodb_2.4.5/slave2", port: 58000, slave: true, source: "localhost:38000" }
Sat Jul 27 12:08:55.069 [replslave] build index local.me { _id: 1 }
Sat Jul 27 12:08:55.070 [replslave] build index done.  scanned 0 total records. 0 secs



3. REPL로 작업하고 확인하기

  - 연결 : mongo <주소:port>

////////////////// slave 1번

mongodb_2.4.5> mongo  localhost:48000
MongoDB shell version: 2.4.5
connecting to: localhost:48000/test
> db.printSlaveReplicationInfo()
source:   localhost:38000
     syncedTo: Sat Jul 27 2013 12:09:53 GMT+0900 (KST)
         = 74 secs ago (0.02hrs)
> ^C
bye

////////////////// slave 2번
mongodb_2.4.5> mongo  localhost:58000
MongoDB shell version: 2.4.5
connecting to: localhost:58000/test
> db.printSlaveReplicationInfo()
source:   localhost:38000
     syncedTo: Sat Jul 27 2013 12:08:53 GMT+0900 (KST)
         = 144 secs ago (0.04hrs)
> ^C
bye

////////////////// master
mongodb_2.4.5> mongo  localhost:38000
MongoDB shell version: 2.4.5
connecting to: localhost:38000/test
> db.printSlaveReplicationInfo()
local.sources is empty; is this db a --slave?


  - master에 데이터 넣고 master에서 save한것이 slave로 복제되는지 확인하기

////////////////// master

> use dowon
switched to db dowon
> db.youngsik.save({name:'dowon', age:22});
> db.youngsik.find();
{ "_id" : ObjectId("51f33a81dbed34cf65d102d0"), "name" : "dowon", "age" : 22 }
> show dbs
config    (empty)
dowon    0.203125GB
local    0.328125GB
> ^C
bye

////////////////// slave 1번
mongodb_2.4.5> mongo  localhost:48000
MongoDB shell version: 2.4.5
connecting to: localhost:48000/test
> show dbs
dowon    0.203125GB
local    0.078125GB
> use dowon
switched to db dowon
> show collections
system.indexes
youngsik
> db.youngsik.find();
{ "_id" : ObjectId("51f33a81dbed34cf65d102d0"), "name" : "dowon", "age" : 22 }
>

////////////////// slave 2번

mongodb_2.4.5> mongo  localhost:58000
MongoDB shell version: 2.4.5
connecting to: localhost:58000/test
> use dowon
switched to db dowon
> show collections
system.indexes
youngsik
> db.youngsik.find();
{ "_id" : ObjectId("51f33a81dbed34cf65d102d0"), "name" : "dowon", "age" : 22 }


  - 파일 시스템 확인하기 : master, slave, slave2

mongodb_2.4.5/slave2>

-rwxr-xr-x   1 nulpulum  staff          6  7 27 12:08 mongod.lock
drwxr-xr-x   4 nulpulum  staff        136  7 27 12:12 journal
-rw-------   1 nulpulum  staff  134217728  7 27 12:12 dowon.1
drwxr-xr-x   2 nulpulum  staff         68  7 27 12:12 _tmp
-rw-------   1 nulpulum  staff   16777216  7 27 12:12 local.ns
-rw-------   1 nulpulum  staff   16777216  7 27 12:12 dowon.ns
-rw-------   1 nulpulum  staff   67108864  7 27 12:12 dowon.0
-rw-------   1 nulpulum  staff   67108864  7 27 12:21 local.0

mongodb_2.4.5/slave>
-rwxr-xr-x   1 nulpulum  staff          6  7 27 12:04 mongod.lock
drwxr-xr-x   4 nulpulum  staff        136  7 27 12:12 journal
-rw-------   1 nulpulum  staff  134217728  7 27 12:12 dowon.1
drwxr-xr-x   2 nulpulum  staff         68  7 27 12:12 _tmp
-rw-------   1 nulpulum  staff   16777216  7 27 12:12 local.ns
-rw-------   1 nulpulum  staff   16777216  7 27 12:13 dowon.ns
-rw-------   1 nulpulum  staff   67108864  7 27 12:13 dowon.0
-rw-------   1 nulpulum  staff   67108864  7 27 12:23 local.0


mongodb_2.4.5/master>
-rwxr-xr-x   1 nulpulum  staff          6  7 27 11:50 mongod.lock
drwxr-xr-x   4 nulpulum  staff        136  7 27 11:51 journal
-rw-------   1 nulpulum  staff   67108864  7 27 12:05 local.0
-rw-------   1 nulpulum  staff  134217728  7 27 12:12 dowon.1
drwxr-xr-x   2 nulpulum  staff         68  7 27 12:12 _tmp
-rw-------   1 nulpulum  staff   16777216  7 27 12:12 dowon.ns
-rw-------   1 nulpulum  staff   67108864  7 27 12:12 dowon.0
-rw-------   1 nulpulum  staff   16777216  7 27 12:22 local.ns
-rw-------   1 nulpulum  staff  268435456  7 27 12:22 local.1



<참조>

  - 10Gen Master/Slave 메뉴얼

posted by 윤영식
2013. 7. 27. 11:18 MongoDB/Prototyping

몽고디비에서 인덱스를 만들고 사용하는 간단한 방법에 대해 실습해 본다.


1. 인덱스 만들기

  - <collection>.ensureIndex({key:1 or -1})   예) db.things.ensureIndex({age:1});

  - 1 (양수) : ascending,  -1 (음수) : descending

> use indextest
switched to db indextest
> db.things.save({age:12, name:'dowon', title:'hi'});
> db.things.save({age:22, name:'dowon2', title:'hi2'});
> db.things.ensureIndex({age:1});
> db.system.indexes.find();
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "indextest.things", "name" : "_id_" }
{ "v" : 1, "key" : { "age" : 1 }, "ns" : "indextest.things", "name" : "age_1" }



2. 인덱스 삭제하기

  - <collection>.dropIndexes({key:1 or -1})  예) db.things.dropIndexes({age:1});

> db.things.dropIndexes({age:1});
{
    "nIndexesWas" : 3,
    "msg" : "non-_id indexes dropped for collection",
    "ok" : 1
}
> db.system.indexes.find();
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "indextest.things", "name" : "_id_" }



3. 인덱스 종류

  - 복합 인덱스 (Compound Key Index) : 여러개의 key를 기반으로 인데스 생성가능

     예) db.things.ensureIndex({age:1, name:-1});

  - 희소 인덱스 (Sparse Index) : 색인된 필드만 인덱스 한다

     예) db.things.ensureIndex({title:1}, {sparse:true}); 기본은 false

  - Unique 인덱스 : 색인된 키에 대해 이미 있거나 중복된 것은 insert되지 않는다

     예) db.things.ensureIndex({name:1}, {unique: true});

  * 하단 참조 Link



<참조>

  - 10Gen의 인덱스 한글 버전

posted by 윤영식
2013. 7. 10. 15:42 MongoDB/Prototyping

Mongoose를 사용할 경우 몇가지 유용한 API를 제공하고 있다. 이중 virtual, method, pre에 대한 사용법을 알아보자. 예제는 로그인시 패스워드를 암호화 하는 코드를 가지고 파악해 본다



1. Schemas

  - http://mongoosejs.com/docs/guide.html 의 Schemas에 대한 정의와 플러그인 확장이 가능하다

  - 스키마 인스턴스를 만들었다면 여기에 methods, statics, indexes, virtuals, options등을 설정할 수 있다

  - 플로그인들은 http://plugins.mongoose.com에서 검색가능하다. 현재 나는 _id에 대한 auto increase 기능을 잘 쓰고 있다 

// 1) 스키마 만들기 : 도큐먼트 구조의 정의

var ComponentSchema = new mongoose.Schema({
    _id         : { type: Number },
    name        : { type: String },
    type        : { type: String },
    config      : { type: String },
    updated_on  : { type: Date, default: Date.now }
  });

  // 2) 플러그인 확장 & methods, statics, indexes, virtuals, options등 설정
  ComponentSchema.plugin(autoinc.plugin, {
    model : 'sd_component',
    field : '_id',
    start : 200000,
    step  : 1
  });


// 3) 모델은 스키마 정의에 대한 컴파일된 생성자이다. Model 인스턴스를 통해서 몽고디비로부터 데이터를 CRUD한다

var Component = mongoose.model('sd_component', ComponentSchema);


  - 샘플로 UserSchema 생성하기

  var UserSchema = new mongoose.Schema({
    username : {type:String, required:true , unique:true},
    hashed_password : {type:String, required:true},
    salt : {type:String, required:true}
  });



2. Instance Methods

  - 스키마가 만들어지고 난후 기본적으로 설정된 메소드를 상속받아 사용한다 예) findById

  - 사용자 커스텀 메소드를 정의할 수 있다. 처리 결과를 return 한다.

  - statics는 callback을 파라미터로 넘기고, return이 없다. (참조)

// 메소드안에서 메소드 호출이 가능하다  

UserSchema.method('authenticate', function(plainText) {
    console.log('authenticate called:')
    console.log('plain text = ' + plainText)
    console.log('hashed = ' + this.encryptPassword(plainText))
    console.log('db password= ' + this.hashed_password)
    return this.encryptPassword(plainText) === this.hashed_password;
  });

  UserSchema.method('makeSalt', function() {
    return Math.round((new Date().valueOf() * Math.random())) + '';
  });


  // crypto 모듈 사용하여 암호화  

  UserSchema.method('encryptPassword', function(password) {
    return crypto.createHmac('sha1', this.salt).update(password).digest('hex');
  });

  UserSchema.method('generateToken', function() {
    return crypto.createHash('md5').update(this.username + Date().toString()).digest("hex");
  });



3. Virtual

  - MongoDB에 저장되지 않는 편리한 Attribute이다 (참조)

  - set/get 펑션을 정의한다

  UserSchema.virtual('password')
    .set(function(password) {
      this._password = password;
      this.salt = this.makeSalt(); // 사용자정의 메소드 호출
      this.hashed_password = this.encryptPassword(password); // 사용자정의 메소드 호출
    })
    .get(function() { return this._password; });


// 모델 생성

var User = mongoose.model('User', UserSchema);

User.password = '1234';     // set

console.log(User.password); // get



4. Pre

  - 몽구스의 middleware기능이다

  - init, validate, save, remove 메소드 수행시 처리되는 미들웨어 펑션이다 

  - 복잡한 유효성검사, 트리거 이벤트 처리등. 예로 사용자를 삭제하면 사용자 관련 블로그포스트도 삭제하기같은 경우 사용

     또는 에러 핸들링

  UserSchema.pre('save', function(next) {
    this.token = this.generateToken();
    if (!validatePresenceOf(this.password || this.hashed_password)) {
      next(new Error('Invalid password'));
    } else {
      next();
    }
  });


function validatePresenceOf(value) {
  return value && value.length;
}


// ex)

User.save(function(err) { console.log(err.message); } );


  - post : 사후처리



5. 전체 소스 코드

  - User Model을 모듈로 만든다.  User.js

  - 애플리케이션에서는 User.js를 Model 인스턴스를 리턴받아서 사용한다 

  - Express의 Middleware에서 사용자 로그인시에 User.authenticate을 호출하여 패스워드를 암호화 한다

var mongoose = require('mongoose'),
  crypto = require('crypto');

module.exports = function () {

  var UserSchema = new mongoose.Schema({
    username : {type:String, required:true , unique:true},
    hashed_password : {type:String, required:true},
    salt : {type:String, required:true}
  });

  UserSchema.virtual('password')
    .set(function(password) {
      this._password = password;
      this.salt = this.makeSalt();
      this.hashed_password = this.encryptPassword(password);
    })
    .get(function() { return this._password; });

  UserSchema.method('authenticate', function(plainText) {
    console.log('authenticate called:')
    console.log('plain text = ' + plainText)
    console.log('hashed = ' + this.encryptPassword(plainText))
    console.log('db password= ' + this.hashed_password)
    return this.encryptPassword(plainText) === this.hashed_password;
  });

  UserSchema.method('makeSalt', function() {
    return Math.round((new Date().valueOf() * Math.random())) + '';
  });

  UserSchema.method('encryptPassword', function(password) {
    return crypto.createHmac('sha1', this.salt).update(password).digest('hex');
  });

  UserSchema.method('generateToken', function() {
    return crypto.createHash('md5').update(this.username + Date().toString()).digest("hex");
  });

  UserSchema.pre('save', function(next) {
    this.token = this.generateToken();
    if (!validatePresenceOf(this.password || this.hashed_password)) {
      next(new Error('Invalid password'));
    } else {
      next();
    }
  });

  return mongoose.model('User', UserSchema);
}

function validatePresenceOf(value) {
  return value && value.length;
}



<참조>

  - mongoosse virtual 사용법

  - Action function과 Callback function의 이해 

  - 샘플소스의 GitHub 저장소

posted by 윤영식
2013. 5. 28. 16:32 MongoDB/Prototyping

Mongoose를 사용하여 개발하는중 외래키를 어떻게 잡는지 궁금하여 찾아보았다. 몽구스 현재 버전은 3.6.11 - GitHub



1. 외래키 잡기 

  - 메뉴얼 : http://mongoosejs.com/docs/populate.html

    + _id가 Number이므로 story에서도 Number로 타입을 가져가야 한다 

var mongoose = require('mongoose') , Schema = mongoose.Schema var personSchema = Schema({ _id : Number, name : String, age : Number, stories : [{ type: Schema.Types.ObjectId, ref: 'Story' }] // 여러 이야기 - 배열 }); var storySchema = Schema({ _creator : { type: Number, ref: 'Person' }, // 글쓴이 한 사람 title : String, fans : [{ type: Number, ref: 'Person' }] // 해당 글에 대한 여러 팬들 - 배열 }); var Story = mongoose.model('Story', storySchema); 

var Person = mongoose.model('Person', personSchema);



2. 플러그인

  - 플로그인은 mongoose의 middleware 기능을 이용하여 만들 수 있다. 즉, 몽고디비에 대한 쿼리가 진행되기 전에 pre action을 수행하고 next할 수가 있는 것이다. Connect 프레임워크의 next와 유사하다고 보면 된다.

  - 플러그인 검색 사이트 : http://plugins.mongoosejs.com

    + 해당 사이트에서 검색을 통하여 플러그인을 찾을 수 있다. 

    + increment로 검색을 수행하여 auto increment 플러그인 찾음.

      https://github.com/mariodu/mongoose-id-autoinc

var dbName = 'id_autoinc_example',
    mongoose = require('mongoose'),
    Schema = mongoose.Schema,
    db = mongoose.createConnection('127.0.0.1', dbName),
// 모듈 로딩
    autoinc = require('../index');

// 모듈 초기화
// Counter 테이블이 만들어 진다. 만일 Sequence 관리용 다른명칭을 주고 싶다면 
// autoinc.init(db, 'AnotherName');
autoinc.init(db);

var UserSchema = new Schema({
  name: String,
  email: String
});

// 플로그인 설정
UserSchema.plugin(autoinc.plugin, {
  model: 'User',
  field: 'seqnumber', // field 정보 안주면 기본 _id 를 Number의 sequence field로 사용한다
  start: 100,
  step: 10
});

var User = db.model('User', UserSchema);

console.log('Database: ' + dbName);
console.log('Collection: ' + User.collection.name);

var user_1 = new User({
  name: 'Dowon',
  email: 'dowon@google.com'
});
user_1.save(function (err, res) {
  console.log('New record added:');
  console.log(res);
});
var user_2 = new User({
  name: 'YoungSik',
  email: 'youngsik@apple.com'
});
user_2.save(function (err, res) {
  console.log('New record added:');
  console.log(res);
  mongoose.disconnect();

});


  - StackOverFlow 에서 도움을 받자 

    + http://stackoverflow.com/questions/tagged/mongoose



3. Smart Dashboard 외래키와 _id 숫자 증가 시키기 

  - service 모듈에서 mongoose와 db connection을 만들어서 전달 받았다. db connection을 두번 만들면 mongoose에서 오류 뱃음.

  - Dashboard안에는 여러 Component가 존재하고 Component는 여러 Dashboard에서 사용 될 수 있다

  - 하기와 같이 사용을 하면 mongo shell 을 통하여 direct insert를 하면 안된다 (주의)


<참조>

  - Connect의 next() 이해하기 (필독)

  - Auto Increment 소스

posted by 윤영식
2013. 2. 16. 16:42 MongoDB/Prototyping

MongoDB 를 접속하기 위한 다양한 방법에 대해 알아보자 



1. mongodb 

  - REST API 개발하기 : 와인 저장고 요약 블로깅

  - MongoDB Native Driver for Node.js 문서

  - 당연히 mongojs 를 사용하는 것보다 복잡하다



2. mongojs

  - mongodb native를 wrapping한 모듈 

  - 설치 : npm install mongojs  

  - mongoDB에서 컬렉션 만들기 

[~/mongodb:nulpulum]./mongo

MongoDB shell version: 2.2.3

connecting to: test

Welcome to the MongoDB shell.

For interactive help, type "help".

For more comprehensive documentation, see

http://docs.mongodb.org/

Questions? Try the support group

http://groups.google.com/group/mongodb-user

> show dbs

local (empty)

> use dowonDB

switched to db dowonDB

> db

dowonDB

> db.people.save({_id:1, age:11, name: 'dowon', sex: true});

> db.people.save({_id:2, age:22, name: 'youngsik', sex: true});

> db.people.save({_id:3, age:33, name: 'taeyoung', sex: false});

> show collections

people

system.indexes


  - mongojs 코딩

  - Mongo Shell 에서 확인

> db.people.find();

{ "_id" : 1, "age" : 11, "name" : "dowon", "sex" : true }

{ "_id" : 2, "age" : 22, "name" : "youngsik", "sex" : true }

{ "_id" : 3, "age" : 33, "name" : "taeyoung", "sex" : false }

{ "_id" : 4, "age" : 27, "name" : "jaein", "sex" : false }



3. mongoose

  - 가장 많이 사용한다. MongoDB접속 ODM(Object Document Mapping) MVC framework

  - 설치하기 : npm install mongoose

  - mongolian 은 mongoose의 light weight 버전이다 

  - Backbone.js 프레임워크와 유사 : schema = controller 개념으로 사용, model = collections

    하기와 같이 schema에서 할 수 있는 일들은 일반 프레임워크의 컨트롤러 즉, 비즈니스 펑션을 제공한다 (참조)

Validators (async and sync)

Defaults

Getters

Setters

Indexes

Middleware

Methods definition

Statics definition

Plugins

pseudo-JOINs

  

  - Schema를 만들 때 사용하는 타입

String

Number

Date

Buffer

Boolean

Mixed

ObjectId

Array

  

  Functional Programming에서 함수를 인자로 넘기고 동일한 로직을 수행한다면, 인자인 함수를 바꿔가면서 동일한 로직을 DRY 하여 사용할 수 있다. 여기서 인자로 넘기는 함수를 Schema라고 생각해 볼 수 있다

  

  - mongoose 코딩

  - 결과 : save, find가 비동기로 이루어지기 때문에 이미 전에 save한 (age: 33 도큐먼트) 데이터만 나온다 

[nulpulum:~/development/mongoose]node mongoose.js 

connection successful...

{ age: 33,

  name: 'dowon',

  sex: true,

  _id: 511f3cb4c6ae3fef03000001,

  __v: 0 }



4. mongolian

  - Node.js 전용 드라이브

  - 향후 다른 layer와 연결하여 개발하려면 mongoose 또는 mongolian/skin중 하나를 선택하여 사용한다 

    (개인적으로 mongoskin framework이 편리하다)



5. mongoskin

  - 설치하기 

[nulpulum:~/development/nodejs/node_basic]npm install mongoskin

npm WARN package.json application-name@0.0.1 No README.md file found!

npm http GET https://registry.npmjs.org/mongoskin

npm http GET https://registry.npmjs.org/mongoskin/-/mongoskin-0.5.0.tgz

npm http GET https://registry.npmjs.org/mongodb

npm http GET https://registry.npmjs.org/bson/0.1.8


> bson@0.1.8 install /Users/nulpulum/development/nodejs/node_basic/node_modules/mongoskin/node_modules/mongodb/node_modules/bson

> (node-gyp rebuild 2> builderror.log) || (exit 0)


xcode-select: Error: No Xcode is selected. Use xcode-select -switch <path-to-xcode>, or see the xcode-select manpage (man xcode-select) for further information.


mongoskin@0.5.0 node_modules/mongoskin

└── mongodb@1.2.13 (bson@0.1.8) 


  - 사용하기 : express 코딩

var mongoskin = require('mongoskin');

var db3 = mongoskin.db('localhost:27017/dowonDB?auto_reconnect');


app.get('/person3/list', function(req, res) {

    console.log('---start 04----');

    db3.collection('people').findOne({name: 'young'}, function (err, users) {

        // do something

        if(err || !users) throw err;


        res.json(users);

    });

});


/////////////////////////////////////////

// 결과 

[nulpulum:~/development/nodejs/node_basic]curl -i -X GET http://localhost:3000/person3/list

HTTP/1.1 200 OK

X-Powered-By: Express

Content-Type: application/json; charset=utf-8

Content-Length: 90

Date: Sat, 23 Feb 2013 05:54:17 GMT

Connection: keep-alive


{

  "age": "31",

  "name": "young",

  "sex": "true",

  "_id": "51282bb231ca135412000001"


  - mongoskin API 



<참조> 

  - Node.js Language Center in MongoDB

  - https://github.com/gett/mongojs

  - MongoJS 사용하기

  - Mongoose API 사용하기

  - Mongoose Homepage

  - Mongoose API 익히자

posted by 윤영식
2013. 1. 5. 16:26 MongoDB/Prototyping

MongoDB에서 Replica Set을 구성하였을 때 mongodb를 접속하는 client에 대하여 routing 서비스를 하는 mongos(몽고스)에 대해 알아보자.



1. 구성 아키텍쳐

  - Shard 한개에 mongod 한개를 맵핑한다

  - config 한개에 mongod 한개만 사용한다



2) 구성하기 

  - shard1, shard2, shard3, config, mongos를 간단히 만들어 보자 (shard를 replica set으로 구성은 배재, 참조)

  - Shard server 3개 띄우기 : sdb01, sdb02, sdb03 디렉토리는 미리 만들어 놓는다. (기동은 개별 콘솔에서 실행함)

$ ./mongod --shardsvr --dbpath /home/dowon/MongoDB/sdb01 --port 10000

Fri Jan  4 22:25:03 

Fri Jan  4 22:25:03 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.

.. 중략 ...

Fri Jan  4 22:25:03 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49

Fri Jan  4 22:25:03 [initandlisten] options: { dbpath: "/home/dowon/MongoDB/sdb01", port: 10000, shardsvr: true }

Fri Jan  4 22:25:03 [websvr] admin web console waiting for connections on port 11000


./mongod --shardsvr --dbpath /home/dowon/MongoDB/sdb02 --port 20000
./mongod --shardsvr --dbpath /home/dowon/MongoDB/sdb03 --port 30000

////////////////////////////
// config srever를 띄웠다. 
$ ./mongod --configsvr --dbpath /home/dowon/MongoDB/sdb04c --port 40000
Fri Jan  4 22:29:47 
Fri Jan  4 22:29:47 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Fri Jan  4 22:29:47 
Fri Jan  4 22:29:47 [initandlisten] MongoDB starting : pid=8201 port=40000 dbpath=/home/dowon/MongoDB/sdb04c 32-bit host=localhost.localdomain
Fri Jan  4 22:29:48 [initandlisten] waiting for connections on port 40000
Fri Jan  4 22:29:48 [websvr] admin web console waiting for connections on port 41000

////////////////////////////
// mongos 띄우기 : chunksize 정하지 않으며 1Mbytes이다. 
$ ./mongos --configdb localhost:40000 --chunkSize 1
Fri Jan  4 22:32:04 warning: running with 1 config server should be done only for testing purposes and is not recommended for production
Fri Jan  4 22:32:04 [mongosMain] MongoS version 2.2.2 starting: pid=8239 port=27017 32-bit host=localhost.localdomain (--help for usage)
Fri Jan  4 22:32:04 [mongosMain] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Fri Jan  4 22:32:04 [mongosMain] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Fri Jan  4 22:32:11 [Balancer] distributed lock 'balancer/localhost.localdomain:27017:1357367525:1804289383' unlocked. 
Fri Jan  4 22:32:17 [Balancer] distributed lock 'balancer/localhost.localdomain:27017:1357367525:1804289383' acquired, ts : 50e7c8f1b1a36d7262b31138


3) mongos 에서 환경 구성하기

  - mongo shell로 들어가서 shard 환경을 만든다 : 자동으로 mongos로 들어간다 (프롬프트 명칭이 mongos가 됨)

$ ./mongo

MongoDB shell version: 2.2.2

connecting to: test

mongos> use admin

switched to db admin

mongos> db

admin

mongos> show dbs

config 0.046875GB

mongos> use peopleDB
switched to db peopleDB
// config 셋팅을 하지 않고 사용하면 다음과 같은 에러가 발생한다. 
mongos> db.people.save({_id:1, age:11, name:'dowon1', sex:false});
Fri Jan  4 22:38:03 uncaught exception: error {
"$err" : "error creating initial database config information :: caused by :: can't find a shard to put new db on",
"code" : 10185
}
mongos> 

/////////////////////////
// 다시 mongo로 들어온다. 
// 그리고 shard에 서버를 추가해 준다.
$ ./mongo
MongoDB shell version: 2.2.2
connecting to: test
mongos> db
test
mongos> use admin
switched to db admin
mongos> db
admin
mongos> db.runCommand({addshard:'localhost:10000'});
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> db.runCommand({addshard:'localhost:20000'});
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> db.runCommand({addshard:'localhost:30000'});
{ "shardAdded" : "shard0002", "ok" : 1 }

////////////////////////////////
// mongos를 통하여 CRUD를 한다
// 과연 입력한 데이터는 Shard가 되었을까? 
mongos> 
mongos> use testShard
switched to db testShard
mongos> db
testShard
mongos> db.people.save({_id:1, age:11, name:'dowon1', sex:false});
mongos> db.people.save({_id:2, age:22, name:'dowon2', sex:true});
mongos> db.people.save({_id:3, age:33, name:'dowon3', sex:false});
mongos> db.people.find();
{ "_id" : 1, "age" : 11, "name" : "dowon1", "sex" : false }
{ "_id" : 2, "age" : 22, "name" : "dowon2", "sex" : true }
{ "_id" : 3, "age" : 33, "name" : "dowon3", "sex" : false }

/////////////////////////////////////
// 결과 확인 : auto shard가 안되어 있다
// sdb01에만 데이터가 저장되어 있다. 이유 : config server를 가동시키지 않았기 때문이다.
[dowon@localhost MongoDB]$ cd sdb01
[dowon@localhost sdb01]$ ll
total 65548
-rwxrwxr-x. 1 dowon dowon        5 Jan  4 22:25 mongod.lock
-rw-------. 1 dowon dowon 16777216 Jan  4 22:43 testShard.0
-rw-------. 1 dowon dowon 33554432 Jan  4 22:42 testShard.1
-rw-------. 1 dowon dowon 16777216 Jan  4 22:43 testShard.ns
drwxrwxr-x. 2 dowon dowon     4096 Jan  4 22:42 _tmp
[dowon@localhost sdb01]$ cd ../sdb02
[dowon@localhost sdb02]$ ll
total 4
-rwxrwxr-x. 1 dowon dowon 5 Jan  4 22:25 mongod.lock
[dowon@localhost sdb02]$ cd ../sdb03
[dowon@localhost sdb03]$ ll
total 4
-rwxrwxr-x. 1 dowon dowon 5 Jan  4 22:25 mongod.lock


4) Shard 테스트를 위한 데이터 만들고 확인하기 

  - shard 연결 환경을 설정한다. (참조 동영상, Shard Guide blog)

////////////////////////////////

// 10000 port mongodb로 들어가자

// 이것은 10000 port의 mongodb로 direct 들어가는 것이다. 

$ ./mongo localhost:10000

MongoDB shell version: 2.2.2

connecting to: localhost:10000/test

> show dbs

local (empty)

testShard 0.0625GB

> use testShard

switched to db testShard

> db

testShard

> show collections

people

system.indexes

> db.people.find();

{ "_id" : 1, "age" : 11, "name" : "dowon1", "sex" : false }

{ "_id" : 2, "age" : 22, "name" : "dowon2", "sex" : true }

{ "_id" : 3, "age" : 33, "name" : "dowon3", "sex" : false }


////////////////////////////
// 20000, 30000 port로 mongos를 통하지 않고 mongodb로 바로 접속하면 
// testShard DB는 없다. 
$ ./mongo localhost:20000
MongoDB shell version: 2.2.2
connecting to: localhost:20000/test
> show dbs
local (empty)

/////////////////////////////////
// mongos로 들어간다. use admin 
// Sharding 환경설정하기
$ ./mongo
MongoDB shell version: 2.2.2
connecting to: test
mongos> dbs
Fri Jan  4 22:54:15 ReferenceError: dbs is not defined (shell):1
mongos> show dbs
config 0.046875GB
testShard 0.0625GB
mongos> use admin
switched to db admin
mongos> db
admin
mongos> db.runCommand({enablesharding:'testShard'});
{ "ok" : 1 }
mongos> db.runCommand({shardcollection:'people', key:{_id:1}});
{ "ok" : 0, "errmsg" : "bad ns[people]" } <-- namespace 에러 
mongos> db.runCommand({shardcollection:'testShard.people', key:{_id:1}});  // _id:1 의 1은 sorting
{ "collectionsharded" : "testShard.people", "ok" : 1 }

//////////////////////////////////////////////
// people collection에 데이터를 넣어보자
mongos> for(var i=6 ; i<100000 ; i++) {
      db.people.save({_id:i, age:44, name:'dowon'+i, sex:true});
   }

//////////////////////////////////////////////////////////////////////////
// 다른 mongo에서 show dbs를 해보면 testShard DB가 생성되었음을 볼 수 있다. 
// chunkSize 1Mbytes 이상이면 sharding이 이루어진다. 
[dowon@localhost mongodb]$ ./mongo localhost:20000
MongoDB shell version: 2.2.2
connecting to: localhost:20000/test
> show dbs
local (empty)
> show dbs
admin (empty)
local (empty)
testShard 0.0625GB
> use testShard
switched to db testShard
> db.people.find();
{ "_id" : 54053, "age" : 44, "name" : "dowon54053", "sex" : true } // 54053 번부터 Sharding 되었다. 
{ "_id" : 54054, "age" : 44, "name" : "dowon54054", "sex" : true }
{ "_id" : 54055, "age" : 44, "name" : "dowon54055", "sex" : true }
{ "_id" : 54056, "age" : 44, "name" : "dowon54056", "sex" : true }
{ "_id" : 54057, "age" : 44, "name" : "dowon54057", "sex" : true }

[dowon@localhost mongodb]$ ./mongo localhost:30000
MongoDB shell version: 2.2.2
connecting to: localhost:30000/test
local (empty)
> show dbs
admin (empty)
local (empty)
testShard 0.0625GB
> use testShard
switched to db testShard
> db.people.find();
{ "_id" : 22200, "age" : 44, "name" : "dowon22200", "sex" : true } // 22200 번부터 Sharding 되었다. 
{ "_id" : 22201, "age" : 44, "name" : "dowon22201", "sex" : true }
{ "_id" : 22202, "age" : 44, "name" : "dowon22202", "sex" : true }
{ "_id" : 22203, "age" : 44, "name" : "dowon22203", "sex" : true }

< 결론 > 
Sharding 시켜놓은 서버로 ChunkSize 크기에 따라 데이터가 분산되어 저장된다. 

 

* 요약 (명령 참조)

  - mongo admin 으로 들어간다

  - runCommand 통하여 세가지 설정을 수행한다. 

  - chunkSize에 따라서 Sharding이 이루어진다.


* Sharding 개념 다시 정리 하기


* MongoDB의 철학은?

데이터와 데이터로직의 분리 

 - 데이터 : JSON (실제 BSON : binary json 형태로 저장됨)

 - 데이터로직 : JavaScript


<참조>

  - 10gen 설정 튜토리얼

posted by 윤영식
2013. 1. 5. 14:56 MongoDB/Prototyping

MongoDB의 Replica Set을 만들어 보자. 먼저 Master/Slave와 Primary/Secondary 순으로 본다. Master/Slave는 Slave에서 Read가 된다. 그러나 Primary/Secondary에서 Secondary는 Read도 되지 않는다.  



1. Replica Set 만들기 위한 Master / Slave 에 대해 알아보자. (Replica Set은 아니다)

  - Master로 DB 수행하기 : master의 mongo에서 CRUD 가능

$ ./mongod -dbpath /home/dowon/MongoDB/db01 --master &

만일 db가 비정상 종료되었다면 

$ ./mongod -dbpath /home/dowon/MongoDB/db01 --master --repair &

수행후에 다시 $ ./mongod -dbpath /home/dowon/MongoDB/db01 --master & 를 수행한다. 


... 하기 메세지 나오면 정상 부팅

Fri Jan  4 19:29:59 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/home/dowon/MongoDB/db01/journal"

Fri Jan  4 19:29:59 [initandlisten] waiting for connections on port 27017

Fri Jan  4 19:29:59 [websvr] admin web console waiting for connections on port 28017

   

  - Slave로 DB 수행하기 : -slave 옵션 

    + slave의 mongo에서 CUD 안됨. Read만 가능하다. 

    + source 지정하여 replica set 만들 master가 누구인지 알려준다. 

    + 기존에 single로 사용하던 mongod를 master/slave로 묶고자 할 경우 기존 데이터를 slave로 복제하기 잘 안됨

       -> 다른 디렉토리로 복사해서 그런가? 테스트 필요함

    + 신규로 master/slave를 띄워서 save 또는 insert 하였을 때 slave 쪽 디렉토리에 db 파일이 제대로 복제됨 

$ ./mongod --dbpath /home/dowon/MongoDB/db02 --port 27018 --slave --source localhost:27017 &

...

Fri Jan  4 19:12:46 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/home/dowon/MongoDB/db02/journal"

Fri Jan  4 19:12:46 [initandlisten] waiting for connections on port 27018   <-- 정상 부팅

Fri Jan  4 19:12:46 [websvr] admin web console waiting for connections on port 28018

// 하기 메세지는 master에 있는 데이터 파일을 slave로 복사해 온다. 마치 git의 pull과 유사한 느낌 ^^ 

Fri Jan  4 19:12:47 [FileAllocator] done allocating datafile /home/dowon/MongoDB/db02/local.ns, size: 16MB,  took 0.001 secs

Fri Jan  4 19:12:48 [FileAllocator] allocating new datafile /home/dowon/MongoDB/db02/local.0, filling with zeroes...

Fri Jan  4 19:12:48 [replslave] build index done.  scanned 0 total records. 0 secs


// db02 디렉토리에 가보면 db01에 있던 데이터베이스 mydb가 copy되어 있는 것을 볼 수 있다. 

Fri Jan  4 19:12:48 [replslave] resync: dropping database mydb

Fri Jan  4 19:12:48 [replslave] resync: cloning database mydb to get an initial copy

Fri Jan  4 19:12:48 [FileAllocator] allocating new datafile /home/dowon/MongoDB/db02/mydb.ns, filling with zeroes...

Fri Jan  4 19:12:48 [FileAllocator] done allocating datafile /home/dowon/MongoDB/db02/mydb.ns, size: 16MB,  took 0.045 secs

Fri Jan  4 19:12:49 [replslave] resync: done with initial clone for db: mydb

Fri Jan  4 19:12:49 [FileAllocator] done allocating datafile /home/dowon/MongoDB/db02/mydb.1, size: 32MB,  took 0.085 secs



2. Replica Set default 만들기 

  - replica set 구동하기 : db11, db12, db13 3개를 띄운다. replica set 명칭은 dowon01 이다. 

  - 디렉토리를 3개 만들고 각 디렉토리별로 서로 다른 port로 replica set 1개를 만들어 본다.

  - mongod를 shutdown 시키고 싶을 경우 : mongo 쉘에서 db.shutdownServer({timeoutSecs: 60});  - 메뉴얼

  - 결론 : mongod를 여러개 띄워서 파일시스템의 약점을 극복한다 (High Availability - HA 구성)

//////////////////////////////////////

// Replica Set 을 1개 만든다 

$ ./mongod --replSet dowon01 --port 30000 --dbpath /home/dowon/MongoDB/db11

$ ./mongod --replSet dowon01 --port 40000 --dbpath /home/dowon/MongoDB/db12

$ ./mongod --replSet dowon01 --port 50000 --dbpath /home/dowon/MongoDB/db13


// 하기 환경설정관련 에러가 나온다 (정상 - 아직 환경설정 안했기 때문에 ^^)

... 중략 ...

Fri Jan 25 23:14:06 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

Fri Jan 25 23:14:23 [initandlisten] connection accepted from 127.0.0.1:34440 #1 (1 connection now open)

Fri Jan 25 23:14:26 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)


/////////////////////////////////////

// mongo client 접속 하기

[dowon@localhost mongodb]$ ./mongo localhost:30000

MongoDB shell version: 2.2.2

connecting to: localhost:30000/test

> show dbs

local (empty)

> exit

bye

[dowon@localhost mongodb]$ ./mongo localhost:40000

MongoDB shell version: 2.2.2

connecting to: localhost:40000/test

> show dbs

local (empty)

> exit

bye

[dowon@localhost mongodb]$ ./mongo localhost:50000

MongoDB shell version: 2.2.2

connecting to: localhost:50000/test

> show dbs

local (empty)


  - Configuration 작업하기 

///////////////////////////////////////////////////

// PRIMARY + SECONDARY Replica Set 만들기 

[dowon@localhost mongodb]$ ./mongo localhost:30000

MongoDB shell version: 2.2.2

connecting to: localhost:30000/test

> var config={_id:'dowon01', members:[{_id:0, host:30000},

... {_id:1, host:40000}^C


// 환경 객체를 만든다 

var config={_id:'dowon01', members:[

    {_id:0, host:'localhost:30000'},

    {_id:1, host:'localhost:40000'},

    {_id:2, host:'localhost:50000'}]

  }


// replica set을 초기화 한다. 

> rs.initiate(config);

{

"info" : "Config now saved locally.  Should come online in about a minute.",

"ok" : 1

}


// 3000 port의 메세지를 보면 다음과 같다. 

// 1) config 없을 때 나오는 메세지이다. 

Fri Jan  4 21:20:29 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

Fri Jan  4 21:20:39 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

// 2) rs.initiate를 수행하면 하기와 같은 메세지가 나온다. 

Fri Jan  4 21:20:44 [conn2] replSet replSetInitiate admin command received from client

Fri Jan  4 21:20:44 [conn2] replSet replSetInitiate config object parses ok, 3 members specified

Fri Jan  4 21:20:44 [conn2] replSet replSetInitiate all members seem up

Fri Jan  4 21:20:44 [conn2] ******

Fri Jan  4 21:20:44 [conn2] replSet info saving a newer config version to local.system.replset

Fri Jan  4 21:20:44 [conn2] replSet saveConfigLocally done

Fri Jan  4 21:20:44 [conn2] replSet replSetInitiate config now saved locally.  Should come online in about a minute.

Fri Jan  4 21:20:44 [conn2] command admin.$cmd command: { replSetInitiate: { _id: "dowon01", members: [ { _id: 0.0, host: "localhost:30000" }, { _id: 1.0, host: "localhost:40000" }, { _id: 2.0, host: "localhost:50000" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:431969 reslen:112 477ms

Fri Jan  4 21:20:49 [rsStart] replSet I am localhost:30000

Fri Jan  4 21:20:49 [rsStart] replSet STARTUP2

Fri Jan  4 21:20:49 [rsHealthPoll] replSet member localhost:40000 is up

Fri Jan  4 21:20:49 [rsHealthPoll] replSet member localhost:50000 is up

Fri Jan  4 21:20:50 [initandlisten] connection accepted from 127.0.0.1:43880 #3 (2 connections now open)

Fri Jan  4 21:20:50 [rsSync] replSet SECONDARY

... 중략 ...
>                                   <== enter key를 치면 하기와 같은 프롬프트로 자동 변경된다
dowon01:PRIMARY>   <== "replicaSet명칭:PRIMARY>" 명칭이 나온다 

////////////////////////////////////////////
// 40000, 50000 번의 client를 들어가 보자 
[dowon@localhost mongodb]$ ./mongo localhost:40000
MongoDB shell version: 2.2.2
connecting to: localhost:40000/test
dowon01:SECONDARY> 

[dowon@localhost mongodb]$ ./mongo localhost:50000
MongoDB shell version: 2.2.2
connecting to: localhost:50000/test
dowon01:SECONDARY> 

//////////////////////////////////////////
// SECONDARY 에서 하기와같은 에러 발생시 rs.slaveOk() 수행
[mongodb@localhost ~]$ ./mongo localhost:40000
MongoDB shell version: 2.2.2
connecting to: localhost:20000/test
dowon01:SECONDARY> show dbs
dowonDB 0.0625GB
local 0.125GB
test (empty)
dowon01:SECONDARY> use dowonDB
switched to db dowonDB
dowon01:SECONDARY> db
dowonDB
dowon01:SECONDARY> show collections
Fri Jan 25 23:45:06 uncaught exception: error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
// primary 의 데이터를 가져온다
dowon01:SECONDARY> rs.slaveOk()
dowon01:SECONDARY> show collections
test
system.indexes

//////////////////////////////////////////
// PRIMARY 에서 데이터베이스 생성하기 
dowon01:PRIMARY> db
test
dowon01:PRIMARY> use peopleDB
switched to db peopleDB
dowon01:PRIMARY> db.people.insert({_id:1, age:11, name:'dowon1', sex:false});
dowon01:PRIMARY> db.people.insert({_id:2, age:22, name:'dowon2', sex:true});
dowon01:PRIMARY> db.people.insert({_id:3, age:33, name:'dowon3', sex:false});
dowon01:PRIMARY> db.people.find();
{ "_id" : 1, "age" : 11, "name" : "dowon1", "sex" : false }
{ "_id" : 2, "age" : 22, "name" : "dowon2", "sex" : true }
{ "_id" : 3, "age" : 33, "name" : "dowon3", "sex" : false }

////////////////////////////////////////////////////////////////////////
// SECONDARY에서 데이터베이스를 READ해 보면 에러가 발생한다
// Replica Set의 Secondary는 Slave가 아니기 때문에 사용할 수 없다. 
// 즉, Primary가 down 되어야 사용 가능하다.
// 단, 1)의 Master/Slave 구조에서 Slave는 READ가 가능하다.
[dowon@localhost mongodb]$ ./mongo localhost:50000
MongoDB shell version: 2.2.2
connecting to: localhost:50000/test
dowon01:SECONDARY> show dbs  // 데이터베이스 보기 
local 0.125GB
peopleDB 0.0625GB
dowon01:SECONDARY> use peopleDB // 해당 데이터베이스로 switching하여 사용하기 
switched to db peopleDB
dowon01:SECONDARY> show collections // 해당 데이터베이스의 테이블(?) 보기 
Fri Jan  4 21:31:59 uncaught exception: error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
dowon01:SECONDARY> db
peopleDB
dowon01:SECONDARY> db.people.find();  // "db"는 peopleDB를 가르키는 포인터
error: { "$err" : "not master and slaveOk=false", "code" : 13435 }


위 명령의 구조는 다음과 같다. (참조 19page)



  - Primary를 종료하면 Sencodary가 Primary로 자동 변경된다

[dowon@localhost mongodb]$ ./mongo localhost:40000

MongoDB shell version: 2.2.2

connecting to: localhost:40000/test

... 중략

dowon01:SECONDARY> show collections

Fri Jan  4 21:31:33 uncaught exception: error: { "$err" : "not master and slaveOk=false", "code" : 13435 }

dowon01:SECONDARY> 

// 30000 port mongoDB를 down시키고 4000 port의 mongo client에서 enter를 치면 하기와 같이 primary로 바뀐다. 

dowon01:PRIMARY> 

// Primary 이므로 CRUD가 가능해 진다. 

dowon01:PRIMARY> show collections
people
system.indexes
dowon01:PRIMARY> db.people.insert({_id:4, age:44, name:'youngsik', sex:true});
dowon01:PRIMARY> db.people.find();
{ "_id" : 1, "age" : 11, "name" : "dowon1", "sex" : false }
{ "_id" : 2, "age" : 22, "name" : "dowon2", "sex" : true }
{ "_id" : 3, "age" : 33, "name" : "dowon3", "sex" : false }
{ "_id" : 4, "age" : 44, "name" : "youngsik", "sex" : true }

/////////////////////////////////////////////////////////////////////
// 다시 30000 port를 기동시키고, 3000 port client로 접속한다.
// 40000 port가 그대로 Primary로 존재하고, 30000 port는 Secondary로 된다. 
i Jan  4 21:42:46 trying reconnect to localhost:30000
Fri Jan  4 21:42:46 reconnect localhost:30000 ok
dowon01:SECONDARY> 



3) Replca Set 기본과 Master/Slave를 결합한 형태

   * 참조 20page




< 참조 > 

- High Availabiltity & Replica Sets with mongoDB

http://www.slideshare.net/shaolintiger/high-availabiltity-replica-sets-with-mongodb


- Sharding by mongodb/10gen on Nov 28, 2012

http://www.slideshare.net/mongodb/sharding-15392976


- Sharding morning session

http://www.slideshare.net/mongodb/sharding-morning-session


- Sharding - Seoul 2012

http://www.slideshare.net/mongodb/sharding-seoul-2012-15576057


- Replication and Replica Sets

http://www.slideshare.net/mongodb/replication-and-replica-sets-tokyo-2012


- MongoDB Basic Concepts

http://www.slideshare.net/mongodb/mongodb-basic-concepts-15674838


- A Morning with MongoDB - Helsinki

http://www.slideshare.net/mongodb/morning-with-mongo-db-helsinki-1


- MongoDB Roadmap

http://www.slideshare.net/mongodb/next-15674517

posted by 윤영식
prev 1 next