MyException - 我的异常网
当前位置:我的异常网» 开源软件 » Hadoop瀛︿範一锛堣浆杞斤級

Hadoop瀛︿範一锛堣浆杞斤級

www.MyException.Cn  网友分享于:2013-02-20  浏览:5次
Hadoop瀛︿範1锛堣浆杞斤級

浠€涔堟槸Hadoop锛?/span>

Hadoop妗嗘灦涓渶鏍稿績鐨勮璁″氨鏄細MapReduce鍜孒DFS銆?/span>MapReduce鐨勬€濇兂鏄敱Google鐨勪竴绡囪鏂囨墍鎻愬強鑰岃骞夸负娴佷紶鐨勶紝绠€鍗曠殑涓€鍙ヨ瘽瑙i噴MapReduce灏辨槸鈥滀换鍔$殑鍒嗚В涓庣粨鏋滅殑姹囨€烩€濄€侶DFS鏄疕adoop鍒嗗竷寮忔枃浠剁郴缁燂紙Hadoop Distributed File System锛夌殑缂╁啓锛屼负鍒嗗竷寮忚绠楀瓨鍌ㄦ彁渚涗簡搴曞眰鏀寔銆?/span>

MapReduce浠庡畠鍚嶅瓧涓婃潵鐪嬪氨澶ц嚧鍙互鐪嬪嚭涓紭鐢憋紝涓や釜鍔ㄨ瘝Map鍜孯educe锛屸€淢ap锛堝睍寮€锛夆€濆氨鏄皢涓€涓换鍔″垎瑙f垚涓哄涓换鍔★紝鈥淩educe鈥濆氨鏄皢鍒嗚В鍚庡浠诲姟澶勭悊鐨勭粨鏋滄眹鎬昏捣鏉ワ紝寰楀嚭鏈€鍚庣殑鍒嗘瀽缁撴灉銆?/span>杩欎笉鏄粈涔堟柊鎬濇兂锛屽叾瀹炲湪鍓嶉潰鎻愬埌鐨勫绾跨▼锛屽浠诲姟鐨勮璁″氨鍙互鎵惧埌杩欑鎬濇兂鐨勫奖瀛愩€備笉璁烘槸鐜板疄绀句細锛岃繕鏄湪绋嬪簭璁捐涓紝涓€椤瑰伐浣滃線寰€鍙互琚媶鍒嗘垚涓哄涓换鍔★紝浠诲姟涔嬮棿鐨勫叧绯诲彲浠ュ垎涓轰袱绉嶏細涓€绉嶆槸涓嶇浉鍏崇殑浠诲姟锛屽彲浠ュ苟琛屾墽琛岋紱鍙︿竴绉嶆槸浠诲姟涔嬮棿鏈夌浉浜掔殑渚濊禆锛屽厛鍚庨『搴忎笉鑳藉棰犲€掞紝杩欑被浠诲姟鏄棤娉曞苟琛屽鐞嗙殑銆傚洖鍒板ぇ瀛︽椂鏈燂紝鏁欐巿涓婅鏃惰澶у鍘诲垎鏋愬叧閿矾寰勶紝鏃犻潪灏辨槸鎵炬渶鐪佹椂鐨勪换鍔″垎瑙f墽琛屾柟寮忋€傚湪鍒嗗竷寮忕郴缁熶腑锛屾満鍣ㄩ泦缇ゅ氨鍙互鐪嬩綔纭欢璧勬簮姹狅紝灏嗗苟琛岀殑浠诲姟鎷嗗垎锛岀劧鍚庝氦鐢辨瘡涓€涓┖闂叉満鍣ㄨ祫婧愬幓澶勭悊锛岃兘澶熸瀬澶у湴鎻愰珮璁$畻鏁堢巼锛屽悓鏃惰繖绉嶈祫婧愭棤鍏虫€э紝瀵逛簬璁$畻闆嗙兢鐨勬墿灞曟棤鐤戞彁渚涗簡鏈€濂界殑璁捐淇濊瘉銆傦紙鍏跺疄鎴戜竴鐩磋涓篐adoop鐨勫崱閫氬浘鏍囦笉搴旇鏄竴涓皬璞★紝搴旇鏄殏铓侊紝鍒嗗竷寮忚绠楀氨濂芥瘮铓傝殎鍚冨ぇ璞★紝寤変环鐨勬満鍣ㄧ兢鍙互鍖规晫浠讳綍楂樻€ц兘鐨勮绠楁満锛岀旱鍚戞墿灞曠殑鏇茬嚎濮嬬粓鏁屼笉杩囨í鍚戞墿灞曠殑鏂滅嚎锛夈€備换鍔″垎瑙e鐞嗕互鍚庯紝閭e氨闇€瑕佸皢澶勭悊浠ュ悗鐨勭粨鏋滃啀姹囨€昏捣鏉ワ紝杩欏氨鏄疪educe瑕佸仛鐨勫伐浣溿€?/span>


鍥?锛歁apReduce缁撴瀯绀烘剰鍥?/span>

涓婂浘灏辨槸MapReduce澶ц嚧鐨勭粨鏋勫浘锛屽湪Map鍓嶈繕鍙兘浼氬杈撳叆鐨勬暟鎹湁Split锛堝垎鍓诧級鐨勮繃绋嬶紝淇濊瘉浠诲姟骞惰鏁堢巼锛屽湪Map涔嬪悗杩樹細鏈塖huffle锛堟贩鍚堬級鐨勮繃绋嬶紝瀵逛簬鎻愰珮Reduce鐨勬晥鐜囦互鍙婂噺灏忔暟鎹紶杈撶殑鍘嬪姏鏈夊緢澶х殑甯姪銆傚悗闈細鍏蜂綋鎻愬強杩欎簺閮ㄥ垎鐨勭粏鑺傘€?/span>

HDFS鏄垎甯冨紡璁$畻鐨勫瓨鍌ㄥ熀鐭筹紝Hadoop鐨勫垎甯冨紡鏂囦欢绯荤粺鍜屽叾浠栧垎甯冨紡鏂囦欢绯荤粺鏈夊緢澶氱被浼肩殑鐗硅川銆傚垎甯冨紡鏂囦欢绯荤粺鍩烘湰鐨勫嚑涓壒鐐癸細

  1. 瀵逛簬鏁翠釜闆嗙兢鏈夊崟涓€鐨勫懡鍚嶇┖闂淬€?/span>
  2. 鏁版嵁涓€鑷存€с€?span style="background-color: #c0c0c0;">閫傚悎涓€娆″啓鍏ュ娆¤鍙栫殑妯″瀷锛屽鎴风鍦ㄦ枃浠舵病鏈夎鎴愬姛鍒涘缓涔嬪墠鏃犳硶鐪嬪埌鏂囦欢瀛樺湪銆?/span>
  3. 鏂囦欢浼氳鍒嗗壊鎴愬涓枃浠跺潡锛屾瘡涓枃浠跺潡琚垎閰嶅瓨鍌ㄥ埌鏁版嵁鑺傜偣涓婏紝鑰屼笖鏍规嵁閰嶇疆浼氱敱澶嶅埗鏂囦欢鍧楁潵淇濊瘉鏁版嵁鐨勫畨鍏ㄦ€с€?/span>


鍥?锛欻DFS缁撴瀯绀烘剰鍥?/span>

涓婂浘涓睍鐜颁簡鏁翠釜HDFS涓変釜閲嶈瑙掕壊锛歂ameNode銆丏ataNode鍜孋lient銆?/span>NameNode鍙互鐪嬩綔鏄垎甯冨紡鏂囦欢绯荤粺涓殑绠$悊鑰咃紝涓昏璐熻矗绠$悊鏂囦欢绯荤粺鐨勫懡鍚嶇┖闂淬€侀泦缇ら厤缃俊鎭拰瀛樺偍鍧楃殑澶嶅埗绛夈€侼ameNode浼氬皢鏂囦欢绯荤粺鐨凪eta-data瀛樺偍鍦ㄥ唴瀛樹腑锛岃繖浜涗俊鎭富瑕佸寘鎷簡鏂囦欢淇℃伅銆佹瘡涓€涓枃浠跺搴旂殑鏂囦欢鍧楃殑淇℃伅鍜屾瘡涓€涓枃浠跺潡鍦―ataNode鐨勪俊鎭瓑銆侱ataNode鏄枃浠跺瓨鍌ㄧ殑鍩烘湰鍗曞厓锛屽畠灏咮lock瀛樺偍鍦ㄦ湰鍦版枃浠剁郴缁熶腑锛屼繚瀛樹簡Block鐨凪eta-data锛屽悓鏃跺懆鏈熸€у湴灏嗘墍鏈夊瓨鍦ㄧ殑Block淇℃伅鍙戦€佺粰NameNode銆侰lient灏辨槸闇€瑕佽幏鍙栧垎甯冨紡鏂囦欢绯荤粺鏂囦欢鐨勫簲鐢ㄧ▼搴忋€傝繖閲岄€氳繃涓変釜鎿嶄綔鏉ヨ鏄庝粬浠箣闂寸殑浜や簰鍏崇郴銆?/span>

鏂囦欢鍐欏叆锛?/span>

  1. Client鍚慛ameNode鍙戣捣鏂囦欢鍐欏叆鐨勮姹傘€?/span>
  2. NameNode鏍规嵁鏂囦欢澶у皬鍜屾枃浠跺潡閰嶇疆鎯呭喌锛岃繑鍥炵粰Client瀹冩墍绠$悊閮ㄥ垎DataNode鐨勪俊鎭€?/span>
  3. Client灏嗘枃浠跺垝鍒嗕负澶氫釜Block锛屾牴鎹瓺ataNode鐨勫湴鍧€淇℃伅锛屾寜椤哄簭鍐欏叆鍒版瘡涓€涓狣ataNode鍧椾腑銆?/span>

鏂囦欢璇诲彇锛?/span>

  1. Client鍚慛ameNode鍙戣捣鏂囦欢璇诲彇鐨勮姹傘€?/span>
  2. NameNode杩斿洖鏂囦欢瀛樺偍鐨凞ataNode鐨勪俊鎭€?/span>
  3. Client璇诲彇鏂囦欢淇℃伅銆?/span>

鏂囦欢Block澶嶅埗锛?/span>

  1. NameNode鍙戠幇閮ㄥ垎鏂囦欢鐨凚lock涓嶇鍚堟渶灏忓鍒舵暟鎴栬€呴儴鍒咲ataNode澶辨晥銆?/span>
  2. 閫氱煡DataNode鐩镐簰澶嶅埗Block銆?/span>
  3. DataNode寮€濮嬬洿鎺ョ浉浜掑鍒躲€?/span>

鏈€鍚庡啀璇翠竴涓婬DFS鐨勫嚑涓璁$壒鐐癸紙瀵逛簬妗嗘灦璁捐鍊煎緱鍊熼壌锛夛細

  1. Block鐨勬斁缃細榛樿涓嶉厤缃€備竴涓狟lock浼氭湁涓変唤澶囦唤锛屼竴浠芥斁鍦∟ameNode鎸囧畾鐨凞ataNode锛屽彟涓€浠芥斁鍦ㄤ笌鎸囧畾DataNode闈炲悓涓€Rack涓婄殑DataNode锛屾渶鍚庝竴浠芥斁鍦ㄤ笌鎸囧畾DataNode鍚屼竴Rack涓婄殑DataNode涓娿€傚浠芥棤闈炲氨鏄负浜嗘暟鎹畨鍏紝鑰冭檻鍚屼竴Rack鐨勫け璐ユ儏鍐典互鍙婁笉鍚孯ack涔嬮棿鏁版嵁鎷疯礉鎬ц兘闂灏遍噰鐢ㄨ繖绉嶉厤缃柟寮忋€?/span>
  2. 蹇冭烦妫€娴婦ataNode鐨勫仴搴风姸鍐碉紝濡傛灉鍙戠幇闂灏遍噰鍙栨暟鎹浠界殑鏂瑰紡鏉ヤ繚璇佹暟鎹殑瀹夊叏鎬с€?/span>
  3. 鏁版嵁澶嶅埗锛堝満鏅负DataNode澶辫触銆侀渶瑕佸钩琛ataNode鐨勫瓨鍌ㄥ埄鐢ㄧ巼鍜岄渶瑕佸钩琛ataNode鏁版嵁浜や簰鍘嬪姏绛夋儏鍐碉級锛氳繖閲屽厛璇翠竴涓嬶紝浣跨敤HDFS鐨刡alancer鍛戒护锛屽彲浠ラ厤缃竴涓猅hreshold鏉ュ钩琛℃瘡涓€涓狣ataNode纾佺洏鍒╃敤鐜囥€備緥濡傝缃簡Threshold涓?0%锛岄偅涔堟墽琛宐alancer鍛戒护鐨勬椂鍊欙紝棣栧厛缁熻鎵€鏈塂ataNode鐨勭鐩樺埄鐢ㄧ巼鐨勫潎鍊硷紝鐒跺悗鍒ゆ柇濡傛灉鏌愪竴涓狣ataNode鐨勭鐩樺埄鐢ㄧ巼瓒呰繃杩欎釜鍧囧€糡hreshold浠ヤ笂锛岄偅涔堝皢浼氭妸杩欎釜DataNode鐨刡lock杞Щ鍒扮鐩樺埄鐢ㄧ巼浣庣殑DataNode锛岃繖瀵逛簬鏂拌妭鐐圭殑鍔犲叆鏉ヨ鍗佸垎鏈夌敤銆?/span>
  4. 鏁版嵁浜ら獙锛氶噰鐢–RC32浣滄暟鎹氦楠屻€傚湪鏂囦欢Block鍐欏叆鐨勬椂鍊欓櫎浜嗗啓鍏ユ暟鎹繕浼氬啓鍏ヤ氦楠屼俊鎭紝鍦ㄨ鍙栫殑鏃跺€欓渶瑕佷氦楠屽悗鍐嶈鍏ャ€?/span>
  5. NameNode鏄崟鐐癸細濡傛灉澶辫触鐨勮瘽锛屼换鍔″鐞嗕俊鎭皢浼氱邯褰曞湪鏈湴鏂囦欢绯荤粺鍜岃繙绔殑鏂囦欢绯荤粺涓€?/span>
  6. 鏁版嵁绠¢亾鎬х殑鍐欏叆锛氬綋瀹㈡埛绔鍐欏叆鏂囦欢鍒癉ataNode涓婏紝棣栧厛瀹㈡埛绔鍙栦竴涓狟lock鐒跺悗鍐欏埌绗竴涓狣ataNode涓婏紝鐒跺悗鐢辩涓€涓狣ataNode浼犻€掑埌澶囦唤鐨凞ataNode涓婏紝涓€鐩村埌鎵€鏈夐渶瑕佸啓鍏ヨ繖涓狟lock鐨凬ataNode閮芥垚鍔熷啓鍏ワ紝瀹㈡埛绔墠浼氱户缁紑濮嬪啓涓嬩竴涓狟lock銆?/span>
  7. 瀹夊叏妯″紡锛氬湪鍒嗗竷寮忔枃浠剁郴缁熷惎鍔ㄧ殑鏃跺€欙紝寮€濮嬬殑鏃跺€欎細鏈夊畨鍏ㄦā寮忥紝褰撳垎甯冨紡鏂囦欢绯荤粺澶勪簬瀹夊叏妯″紡鐨勬儏鍐典笅锛屾枃浠剁郴缁熶腑鐨勫唴瀹逛笉鍏佽淇敼涔熶笉鍏佽鍒犻櫎锛岀洿鍒板畨鍏ㄦā寮忕粨鏉熴€傚畨鍏ㄦā寮忎富瑕佹槸涓轰簡绯荤粺鍚姩鐨勬椂鍊欐鏌ュ悇涓狣ataNode涓婃暟鎹潡鐨勬湁鏁堟€э紝鍚屾椂鏍规嵁绛栫暐蹇呰鐨勫鍒舵垨鑰呭垹闄ら儴鍒嗘暟鎹潡銆傝繍琛屾湡閫氳繃鍛戒护涔熷彲浠ヨ繘鍏ュ畨鍏ㄦā寮忋€傚湪瀹炶返杩囩▼涓紝绯荤粺鍚姩鐨勬椂鍊欏幓淇敼鍜屽垹闄ゆ枃浠朵篃浼氭湁瀹夊叏妯″紡涓嶅厑璁镐慨鏀圭殑鍑洪敊鎻愮ず锛屽彧闇€瑕佺瓑寰呬竴浼氬効鍗冲彲銆?/span>

涓嬮潰缁煎悎MapReduce鍜孒DFS鏉ョ湅Hadoop鐨勭粨鏋勶細


鍥?锛欻adoop缁撴瀯绀烘剰鍥?/span>

鍦℉adoop鐨勭郴缁熶腑锛屼細鏈変竴鍙癕aster锛屼富瑕佽礋璐ameNode鐨勫伐浣滀互鍙奐obTracker鐨勫伐浣溿€侸obTracker鐨勪富瑕佽亴璐e氨鏄惎鍔ㄣ€佽窡韪拰璋冨害鍚勪釜Slave鐨勪换鍔℃墽琛屻€傝繕浼氭湁澶氬彴Slave锛屾瘡涓€鍙癝lave閫氬父鍏锋湁DataNode鐨勫姛鑳藉苟璐熻矗TaskTracker鐨勫伐浣溿€俆askTracker鏍规嵁搴旂敤瑕佹眰鏉ョ粨鍚堟湰鍦版暟鎹墽琛孧ap浠诲姟浠ュ強Reduce浠诲姟銆?/span>

璇村埌杩欓噷锛屽氨瑕佹彁鍒板垎甯冨紡璁$畻鏈€閲嶈鐨勪竴涓璁$偣锛歁oving Computation is Cheaper than Moving Data銆傚氨鏄湪鍒嗗竷寮忓鐞嗕腑锛岀Щ鍔ㄦ暟鎹殑浠d环鎬绘槸楂樹簬杞Щ璁$畻鐨勪唬浠枫€傜畝鍗曟潵璇村氨鏄垎鑰屾不涔嬬殑宸ヤ綔锛岄渶瑕佸皢鏁版嵁涔熷垎鑰屽瓨鍌紝鏈湴浠诲姟澶勭悊鏈湴鏁版嵁鐒跺悗褰掓€伙紝杩欐牱鎵嶄細淇濊瘉鍒嗗竷寮忚绠楃殑楂樻晥鎬с€?/span>

涓轰粈涔堣閫夋嫨Hadoop锛?/span>

璇村畬浜哤hat锛岀畝鍗曞湴璇翠竴涓媁hy銆傚畼鏂圭綉绔欏凡缁忕粰浜嗗緢澶氱殑璇存槑锛岃繖閲屽氨澶ц嚧璇翠竴涓嬪叾浼樼偣鍙婁娇鐢ㄧ殑鍦烘櫙锛堟病鏈変笉濂界殑宸ュ叿锛屽彧鐢ㄤ笉閫傜敤鐨勫伐鍏凤紝鍥犳閫夋嫨濂藉満鏅墠鑳藉鐪熸鍙戞尌鍒嗗竷寮忚绠楃殑浣滅敤锛夛細

  1. 鍙墿灞?/span>锛氫笉璁烘槸瀛樺偍鐨勫彲鎵╁睍杩樻槸璁$畻鐨勫彲鎵╁睍閮芥槸Hadoop鐨勮璁℃牴鏈€?/span>
  2. 缁忔祹锛氭鏋跺彲浠ヨ繍琛屽湪浠讳綍鏅€氱殑PC涓娿€?/span>
  3. 鍙潬锛氬垎甯冨紡鏂囦欢绯荤粺鐨勫浠芥仮澶嶆満鍒朵互鍙奙apReduce鐨勪换鍔$洃鎺т繚璇佷簡鍒嗗竷寮忓鐞嗙殑鍙潬鎬с€?/span>
  4. 楂樻晥锛氬垎甯冨紡鏂囦欢绯荤粺鐨勯珮鏁堟暟鎹氦浜掑疄鐜颁互鍙奙apReduce缁撳悎Local Data澶勭悊鐨勬ā寮忥紝涓洪珮鏁堝鐞嗘捣閲忕殑淇℃伅浣滀簡鍩虹鍑嗗銆?/span>

浣跨敤鍦烘櫙锛?/span>涓汉瑙夊緱鏈€閫傚悎鐨勫氨鏄?span style="background-color: #c0c0c0;">娴烽噺鏁版嵁鐨勫垎鏋?/span>锛屽叾瀹濭oogle鏈€鏃╂彁鍑篗apReduce涔熷氨鏄负浜嗘捣閲忔暟鎹垎鏋愩€傚悓鏃禜DFS鏈€鏃╂槸涓轰簡鎼滅储寮曟搸瀹炵幇鑰屽紑鍙戠殑锛屽悗鏉ユ墠琚敤浜庡垎甯冨紡璁$畻妗嗘灦涓€傛捣閲忔暟鎹鍒嗗壊浜庡涓妭鐐癸紝鐒跺悗鐢辨瘡涓€涓妭鐐瑰苟琛岃绠楋紝灏嗗緱鍑虹殑缁撴灉褰掑苟鍒拌緭鍑恒€傚悓鏃剁涓€闃舵鐨勮緭鍑哄張鍙互浣滀负涓嬩竴闃舵璁$畻鐨勮緭鍏ワ紝鍥犳鍙互鎯宠薄鍒颁竴涓爲鐘剁粨鏋勭殑鍒嗗竷寮忚绠楀浘锛屽湪涓嶅悓闃舵閮芥湁涓嶅悓浜у嚭锛屽悓鏃跺苟琛屽拰涓茶缁撳悎鐨勮绠椾篃鍙互寰堝ソ鍦板湪鍒嗗竷寮忛泦缇ょ殑璧勬簮涓嬪緱浠ラ珮鏁堢殑澶勭悊銆?/span>

文章评论

软件开发程序错误异常ExceptionCopyright © 2009-2015 MyException 版权所有