[BUG] Feroxbuster misses existing endpoints (under load?)

[mzember]

Describe the bug
Under specific circumstances, an existing URL is not detected.

To Reproduce
It is much more difficult to reproduce in v2.13.1, I struggle to isolate the test case.

In a course lab (that is not mine), it manifests like this:

1. /api/heartbeat exists.
2. Directory /api/ is discovered, together with other 5 directories. Recursion proceeds into them.
3. Some wanings are displayed (but they do not mention "heartbeat"), auto-tune slows down, then speeds up, etc.

WRN   1126.446 feroxbuster::utils Error while making request: error sending request for url (http://192.168.196.143:443/api/System%20Tools.txt)
WRN   1126.447 feroxbuster::utils err: error sending request for url (http://192.168.196.143:443/api/System%20Tools.txt)
WRN   1126.447 feroxbuster::scanner::ferox_scanner Requester encountered an error: error sending request for url (http://192.168.196.143:443/api/System%20Tools.txt)

4. Feroxbuster passes the points where "heartbeat" is in the wordlist:

kali@kali:~/oscp/chal3-A/143$ grep -n heartbeat /usr/share/seclists/Discovery/Web-Content/concatenation-of-quickhits.txt-raft-medium-directories.txt-directory-list-2.3-medium.txt
27279:heartbeat
72053:heartbeat

5. Despite present, /api/heartbeat is not in the feroxbuster output file neither replayed in Burp.

Additional context

feroxbuster --smart -v  --collect-words --collect-extensions --url=http://192.168.196.143:443/ --burp-replay --wordlist=/usr/share/seclists/Discovery/Web-Content/concatenation-of-quickhits.txt-raft-medium-directories.txt-directory-list-2.3-medium.txt -o ferox443new.out --dont-scan '/%3f$',server-status
 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.13.1
───────────────────────────┬──────────────────────
 🎯  Target Url            │ http://192.168.196.143:443/
 🚫  Don't Scan Regex      │ /%3f$
 🚫  Don't Scan Regex      │ server-status
 🚩  In-Scope Url          │ 192.168.196.143
 🚀  Threads               │ 50
 📖  Wordlist              │ /usr/share/seclists/Discovery/Web-Content/concatenation-of-quickhits.txt-raft-medium-directories.txt-directory-list-2.3-medium.txt
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.13.1
 💉  Config File           │ /etc/feroxbuster/ferox-config.toml
 🎥  Replay Proxy          │ http://127.0.0.1:8080
 📼  Replay Proxy Codes    │ [100, 101, 102, 200, 201, 202, 203, 204, 205, 206, 207, 208, 226, 300, 301, 302, 303, 304, 305, 307, 308, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 421, 422, 423, 424, 426, 428, 429, 431, 451, 500, 501, 502, 503, 504, 505, 506, 507, 508, 510, 511, 103, 425]
 🔎  Extract Links         │ true
 💾  Output File           │ ferox443new.out
 💰  Collect Extensions    │ true
 💸  Ignored Extensions    │ [Images, Movies, Audio, etc...]
 🏦  Collect Backups       │ true
 🤑  Collect Words         │ true
 🏁  HTTP methods          │ [GET]
 🔓  Insecure              │ true
 🎶  Auto Tune             │ true
 🔈  Verbosity             │ 1
 🔃  Recursion Depth       │ 4

[>-------------------] - 19m    10455/3246987 9/s     http://192.168.196.143:443/ => 🚦 reduced scan speed (1/s)
[###>----------------] - 19m   471370/2934116 419/s   http://192.168.196.143:443/api/ => 🚦 reduced scan speed (65/s)
[>-------------------] - 18m    14613/3269267 13/s    http://192.168.196.143:443/plugins/ => 🚦 reduced scan speed (2/s)
[>-------------------] - 18m     3710/3285742 3/s     http://192.168.196.143:443/themes/ => 🚦 increased scan speed (2/s)
[>-------------------] - 18m     5938/3282016 5/s     http://192.168.196.143:443/content/ => 🚦 increased scan speed (1/s)
[>-------------------] - 18m     4672/3285586 4/s     http://192.168.196.143:443/assets/ => 🚦 reduced scan speed (0/s)
[>-------------------] - 18m    63167/3234543 58/s    http://192.168.196.143:443/config/ => 🚦 increased scan speed (11/s)
[>-------------------] - 12m    57409/3266264 80/s    http://192.168.196.143:443/vendor/ => 🚦 reduced scan speed (25/s)

Expected behavior
/api/heartbeat is found

Environment (please complete the following information):

• feroxbuster version: v2.13.1
• OS Kali Rolling Linux

Previous versions
I suspected auto-tune, not retrying URLs that failed, but it is much more robust in 2.13.1. Creating a custom wordlist (according to when auto-tune speeds up) was possible/easier in 2.13.0. Or adding more threads, simulating a bigger load.

[epi052]

howdy!

a few possibilities that come to mind

• the auto-filter found for /api applies to /api/heartbeat and it is filtered out
• the dont-scan regex is preventing the request (unlikely)
• the target changes out from under you (this looks like a challenge box that's likely only you on it, if not, add shared machine shenanigans to the list)
• where heartbeat sits in the wordlist isn't overly concerning. it may get requested out of order (shouldn't be wildly out of order, but it's definitely not serial)
• auto-tune as the culprit is pretty far down on my personal list of possibilities. it shouldn't alter what's sent/received, only the pace at which requests are made

have you tried with a small/targeted wordlist on only the /api endpoint? does it ever fail? does it fail sometimes?

[mzember]

thank you for the insights.

I have tried with a small wordlist again, and raising the number of threads (over the edge when it works reliably).

I checked logs and output files and I was wrong in the original report, the desired path was actually discovered (isn't that great!?)

If I raise the number of threads (extremely, to 2000), it manifests sometimes.

$ feroxbuster --smart -v --dont-scan server-status,config --url=http://`cat target`:443/ \
 --wordlist=wordlist-testing -o feroxtesting.out --filter-similar-to http://`cat target`:443/api/xxxxxxxx \
 --threads 2000

I may try to create a webserver that will log everything it receives and then compare it to what is expected to receive.

It is much better with the last update. I suspect the webserver cannot handle the load at some moments.

[mzember]

The end of the output then looks like this:

WRN     10.632 feroxbuster::utils err: error sending request for url (http://192.168.196.143:443/2497)
WRN     10.632 feroxbuster::scanner::ferox_scanner Requester encountered an error: error sending request for url (http://192.168.196.143:443/2497)
WRN     10.638 feroxbuster::utils Error while making request: error sending request for url (http://192.168.196.143:443/2498)
WRN     10.638 feroxbuster::utils err: error sending request for url (http://192.168.196.143:443/2498)
WRN     10.639 feroxbuster::scanner::ferox_scanner Requester encountered an error: error sending request for url (http://192.168.196.143:443/2498)
WRN     10.642 feroxbuster::utils Error while making request: error sending request for url (http://192.168.196.143:443/2499)
WRN     10.642 feroxbuster::utils err: error sending request for url (http://192.168.196.143:443/2499)
WRN     10.642 feroxbuster::scanner::ferox_scanner Requester encountered an error: error sending request for url (http://192.168.196.143:443/2499)
WRN     10.649 feroxbuster::utils Error while making request: error sending request for url (http://192.168.196.143:443/2500)
WRN     10.649 feroxbuster::utils err: error sending request for url (http://192.168.196.143:443/2500)
WRN     10.649 feroxbuster::scanner::ferox_scanner Requester encountered an error: error sending request for url (http://192.168.196.143:443/2500)
WRN     11.320 feroxbuster::utils Error while making request: error sending request for url (http://192.168.196.143:443/icons/ubuntu-logo.png~)
WRN     11.320 feroxbuster::event_handlers::outputs Could not request backup of http://192.168.196.143:443/icons/ubuntu-logo.png
200      GET      375l      964w    10918c http://192.168.196.143:443/
[####################] - 11s     2514/2514    0s      found:1       errors:1340   
[####################] - 10s     2502/2502    247/s   http://192.168.196.143:443/   
$ tail feroxtesting.out        
    update_app: false,
    scan_dir_listings: false,
    request_file: "",
    protocol: "https",
    limit_bars: 0,
    unique: false,
    response_size_limit: 4194304,
}
200      GET       15l       74w     6147c http://192.168.196.143:443/icons/ubuntu-logo.png
200      GET      375l      964w    10918c http://192.168.196.143:443/
$ grep 'api\|heart' wordlist-testing -n
1550:api
2476:heartbeat  

I planted the words so that it's their turn when the warnings are appearing.

Recursing into multiple directories seems to cause a big load on the webserver.