Changes between Version 5 and Version 6 of waue/2009/0408

Timestamp:

Apr 8, 2009, 4:12:27 PM (15 years ago)

Author:

waue

Comment:

--

waue/2009/0408

@@ -49 +49 @@
                 <name>plugin.includes</name> <value>protocol-file|protocol-http|parse-(text|html)|index-basic|query-(basic|site|url)</value>
         </property>
+
+
+What is happening?
+By default, the size of the documents downloaded by Nutch is limited (to 65536 bytes). To allow Nutch to download larger files (via HTTP), modify nutch-site.xml and add an entry like this:
+            <property>
+                  <name>http.content.limit</name> <value>150000</value>
+            </property>
+      If you do not want to limit the size of downloaded documents, set http.content.limit to a negative value:
+            <property>
+                  <name>http.content.limit</name> <value>-1</value>
+            </property>
+
+
 }}}