hadoop reducer progress 100% but doesn't end

56
September 13, 2017, at 10:18 PM

I'm running one hadoop job with 16 reduce tasks. (hadoop-1.2.1)

Some of them have 100% progress, but they don't end. they just stuck at 100% and job clean up : pending. And their status displays only 'reduce > reduce'.

I guess this problem occurs because the job writes too many multiple outputs. (about 50~100 multiple outputs) But it's just my guess. I don't know exactly that writing many multiple outputs causes this problem. Could anyone tell me what causes this problem in detail? I searched a lot for this, but I could not find a clear answer.

Best regards.

Answer 1

Have you looked in the task logs? You should be able to tell what the task is doing. For instance, you should see if it is merging files.

Also, make note of how much data the reducers are processing. Are the reducers that are completing processing less data than the ones that are not completing in the same time? If so then this points to reducer skew (data skew) they are doing more work than the others.

READ ALSO
JPA inserting header and detail data

JPA inserting header and detail data

I'm new to JPA I want to insert detail data using entitymanagerpersist() here is the header snippet

68
Reduce Google Cloud Endpoints request response header size

Reduce Google Cloud Endpoints request response header size

I'm using a Google Cloud App Engine and Google Cloud Endpoints API for my backend for an Android appThis is all in Java

74
Spring evaluation language (SpEL) not throw exception when accessing invalid key in map

Spring evaluation language (SpEL) not throw exception when accessing invalid key in map

I am new to SpEL and below is a sample workable test that I used for testing

91
While authenticating mongodb using java it is taking more time and throwing mongotimeoutException in case of wrong credential

While authenticating mongodb using java it is taking more time and throwing mongotimeoutException in case of wrong credential

After authentication i need to show message to end userBut, the exception is comming after 30 seconds in case when the user enters wrong credentials

85