Wednesday, April 3, 2013

Weird Combiner in Hadoop

I write my own MapReduce functions for K means problem.

When I test my code using a small dataset(Small than 1M), it works very well. However, when I apply it to larger one(More than 100M), it returns errors.

Actually, I am not so sure about my idea. But I believe it is true at 90%.


  • When I test small dataset, combiner just be called once between mapper and reducer.

  • However, when I test large dataset, combiner is called more than once between one pair of mapper and reducer.(It is so weird, and I am not so sure, because I haven't study hadoop source code)
You can find my source code for K-Means here:
https://github.com/zhouhao/Hadoop_KMeans_MapReduce_Java/

Attention for strtok() in C



Head file:#include<string.h>
Function prototype: char * strtok(char *s,const char *delim);
Description:divide string s into pieces according to string delim
  1. strtok() scans s, if it finds the current character is in delim(delim is a character set), then strtok() replaces current character with '\0'[this means string s is changed]
  2. At first, strtok() needs s as a parameter, but then, we can call strtok() by set s as NULL, strtok() will return the next string point(If there is no strings any more, strtok() will return NULL).
#include<stdio.h>
#include<string.h>
int main()
{
    char s[]="ab-c656f;gh,i-jkl;mnop;54;fdfdz";
    char *delim="-, ";
    char *p;
    printf("%s ";strtok(s,delim));
    //printf("%s ";s); //s is changed
    while((p=strtok(NULL,delim)))
    {
        printf("%s ",p);
    }
    printf("\n");
    return 0;
}

-----------------------My work for Strtok()-----------------

int numOfParametersCheck(const uchar *str,const int num)
{
    int numParameters=0;
uchar cpyStr[MAXLEN_CMD];
strcpy(cpyStr,str);//I duplicate a new string 
strtok(cpyStr," ");
while(strtok(NULL," ")!=NULL)
{
   numParameters++;
   if(numParameters>num)
   {
       printf("Cmd should have %d parameter!\n\n",num);
            return 0;
   }
}
if(numParameters==num)
{
   return 1;
}
else  //numParameters<num
{
        printf("Cmd should have %d parameter!\n\n",num);
   return 0;
}
}

Monday, April 1, 2013

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt

When I re-install my OS with Win7(X64).
I installed .net FrameWork 4.5 before VS2010. So when I run C(C++)project in VS2010, it occurs with an error like the title.

How to fix it:
Project Properties 
   -> Configuration Properties 
       -> Linker (General) 
          -> Enable Incremental Linking -> "No (/INCREMENTAL:NO)"
Only this way works without re-install anything.


reference:http://stackoverflow.com/questions/10888391/link-fatal-error-lnk1123-failure-during-conversion-to-coff-file-invalid-or-c