bitpt
Karma: 0
|
Multi Downloads Issue - 2010/07/30 15:23
I have remository with more then 18000 files, 35gig in files, in the last days i saw server logs with multi downloads, i have limit in 10 downloads/day.
a few minutes ago i have 25 users download at same time, wich user with more then 1g with more then 1 hour
Remository rip attempt is sucessfull if i don't kill conection from ripper, i try activate licence to stop ripp and scrapper software download, i try a little hack with captcha, unsucessfull.
search the web to find how pass remository download key, i don't find nothing.
a sample a few minutes ago stoped with server ip block....
/index.php/component/option,com_remository/Itemid,34/func,download/id,8146/
chk,9af2d82c6c18ca9cf86ae1460586990a/no_html,1/index.html
this user download +/- 500 files, how it is possible ?
Any ideias to stop this ?
thanks in advance
Post edited by: bitpt, at: 2010/07/30 19:53
|
|
|
| | Sorry, you do not currently have permission to write here. |
admin
Karma: 98
|
Re:Multi Downloads Issue - 2010/07/31 12:12
What version of Remository are you using?
Do you have information on the user agent (UA) presented by the downloader?
Martin Brampton aka Counterpoint
http://aliro.org
http://black-sheep-research.com |
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/03 15:16
I' m using the V3.52.8, Ripper UA is always the firefox 3.6. (seems a add-on, i try all download add-on for firefox don't find any can do this)
after several days of testing, i think it's some software that generates the download key, before the downloads, ripper catch all download links (fileinfo), then start the downloads.
I remembered, if i have a different download key, not possibly download all the files. i add a new string at the end of download key
like $newkey="142535" -> whatever the number
in the end of .date and unix time i put -> .$newkey
Ripping stop for now, i do not know for sure if i can do this without another problems, until now, works fine. Multi download stop even with many attempts.
Thanks
Post edited by: bitpt, at: 2010/08/03 17:56
Post edited by: bitpt, at: 2010/08/03 17:57
|
|
|
| | Sorry, you do not currently have permission to write here. |
admin
Karma: 98
|
Re:Multi Downloads Issue - 2010/08/03 18:05
Are the multiple downloads being recorded in the Remository log table - jos_downloads_log?
Martin Brampton aka Counterpoint
http://aliro.org
http://black-sheep-research.com |
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/03 20:40
Normal downloads yes, for sure.... when rippers came i don't know, i'll check.
Last minute log, i clean it a few days ago....
19838 1 2010-08-03 21:31:45 0 15893 5567184 190.178.117.xxx
19839 1 2010-08-03 21:31:46 0 1640 299945 187.66.217.xxx
19840 1 2010-08-03 21:31:47 0 12194 541179 89.136.179.xxx
19841 1 2010-08-03 21:31:47 0 6826 367558 187.58.153.xxx
19842 1 2010-08-03 21:31:51 0 622 550603 187.21.242.xxx
19843 1 2010-08-03 21:31:55 0 17213 151207 109.188.117.xxx
19844 1 2010-08-03 21:32:16 0 17213 151207 109.188.117.xxx
19845 1 2010-08-03 21:32:21 0 11919 1757170 201.58.9.xxx
19846 1 2010-08-03 21:32:23 0 953 1448990 187.78.55.xxx
19847 1 2010-08-03 21:32:25 0 15896 1132565 187.22.93.xxx
19848 1 2010-08-03 21:32:38 0 12873 2767242 189.46.165.xxx
19849 1 2010-08-03 21:32:39 0 13057 958338 201.34.157.xxx
19850 1 2010-08-03 21:32:42 0 2539 1368803 200.167.42.xxx
19851 1 2010-08-03 21:32:45 0 17192 2049906 188.80.192.xxx
19852 1 2010-08-03 21:32:51 0 954 287196 201.66.109.xxx
19853 1 2010-08-03 21:32:55 0 4353 146420 187.6.217.xxx
19854 1 2010-08-03 21:32:56 0 4847 1016649 200.181.138.xxx
19855 1 2010-08-03 21:32:57 0 953 1448990 187.78.55.xxx
19856 1 2010-08-03 21:32:59 0 4081 7277922 190.178.117.xxx
19857 1 2010-08-03 21:32:59 0 13560 196213 189.16.42.xxx
19858 1 2010-08-03 21:33:04 0 1467 293678 189.87.197.xxx
19859 1 2010-08-03 21:33:04 0 3620 751612 190.122.0.xxx
19860 1 2010-08-03 21:33:04 0 14639 121005 189.112.182.xxx
Post edited by: bitpt, at: 2010/08/03 23:10
|
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/05 20:38
log table haven't scrappers IP or download, a few motnhs ago i stop accelerators to use multiple connections, i stop that in htaccess, download managers, DAP etc etc are blocked, but new browsers IE8, FF3.6 make several conections .
In this case, server logs have multiple connections for a file download, and seems to me, even download is finished, all download give 206 error (partial content download).
In this case table log don't cache ip, we can't limit downloads to that user.
One download in server log.
187.78.71.xxx /index.php/component/option,com_remository/Itemid,34/func,download/id,8188/
chk,dc99d57fa17b5c73000361d5188e116d/no_html,1/index.html 206 475,6 KB (domain)/index.php/component/option,com_remository/Itemid,34/func,startdown/id,8188/
187.78.71.xxx index.php/component/option,com_remository/Itemid,34/func,download/id,8188/
chk,dc99d57fa17b5c73000361d5188e116d/no_html,1/index.html 206 640,0 KB (domain)/index.php/component/option,com_remository/Itemid,34/func,startdown/id,8188/
187.78.71.xxx /index.php/component/option,com_remository/Itemid,34/func,download/id,8188/
chk,dc99d57fa17b5c73000361d5188e116d/no_html,1/index.html 206 640,0 KB (domain)/index.php/component/option,com_remository/Itemid,34/func,startdown/id,8188/
any ideia how to solve and protect this ?
I'm seeing this right ?
Thanks in advance
Sorry my bad English
Post edited by: bitpt, at: 2010/08/05 20:59
|
|
|
| | Sorry, you do not currently have permission to write here. |
admin
Karma: 98
|
Re:Multi Downloads Issue - 2010/08/05 22:42
Remository records downloads in the jos_downloads_log table. If entries are not written there, Remository does not know about the downloads. Log records are written at the completion of a download, because otherwise failed downloads may be logged and legitimate users blocked.
It would need new code for Remository to place limits on the same IP address starting a download while one is still in progress.
Martin Brampton aka Counterpoint
http://aliro.org
http://black-sheep-research.com |
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/05 23:28
with multi download slots download never finished because user get partial parts, Remository never know even if all parts of a file are completed.
In sample before, total file have 1,7M more a less, user download 3 parts, downloaded complete file but system never know, he can download all remository content without limitations.
Can you give me a ideia how to solve this ? it's easy change this in FF
network.http.max-persistent-connections-per-server;6 -> by default
i'll try Mod_limitipconn directly on server...
Post edited by: bitpt, at: 2010/08/06 00:03
|
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/06 12:16
Mod limitipconn on server don't work because server cache is on.
I don't know what to do, i check several sites with remository, all have this issue
any ideas ?
Thank You.
|
|
|
| | Sorry, you do not currently have permission to write here. |
admin
Karma: 98
|
Re:Multi Downloads Issue - 2010/08/08 22:44
It is difficult to know what would count as a solution. Do you have an idea what you need?
The handling of requests for partial downloads could be blocked. That would mean that each request would deliver a whole file, starting from the beginning. But if downloaders are just creating trouble, this would not be effective if they cancelled each download before completion.
It would be possible to count downloads at the start, instead of the finish, but that would penalise legitimate users whose downloads failed.
The total bandwidth could be logged, but that would involve a lot more database activity. Holding the information in a session would not work, as nothing would trigger a save.
Any suggestions?
Martin Brampton aka Counterpoint
http://aliro.org
http://black-sheep-research.com |
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/09 13:16
After several hours of tests in previous days, I noticed that some visitors, wich download files in parts, do not even know who act that way, there are several download managers doing this.
The problem does not seem to be me only, I checked and asked to check several logs.
If remository have few files, no problem, don't affect normal operation of the site, on the other hand, if someone has remository limit, for any reason, is affected because the system does not work out. If you check the logs (remository.com) and database downloads_log, you have dowloads not recorded.
The best way to Remository, i think, in the next versions,
be possible to limit the (log) complete download or (log)start the download (option) and shall not permit or allow multiple downloads (start and finish one, start new only when last finished) optional too.
In my case, the best way is limit the downloads, just start one and until finished no more downloads, save the bandwidth log does not seem to be a good idea, in my case I have files with 100k and 20 megs (it's a electronic site, technical schematics etc etc..). Can you point where de limit code is, i'll try do something.
Or, download limit work, even downloader use partial file download or not.
Thanks for your reply.
Post edited by: bitpt, at: 2010/08/09 16:56
|
|
|
| | Sorry, you do not currently have permission to write here. |
admin
Karma: 98
|
Re:Multi Downloads Issue - 2010/08/10 00:26
Do you have the modified database drivers installed? From http://remository.com/downloads/support-files/joomla-1.5/
Without those, large downloads will not be logged because Joomla does not handle the "database gone away" situation.
Martin Brampton aka Counterpoint
http://aliro.org
http://black-sheep-research.com |
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/10 01:48
I use old 1.0.15 with many modz. In logs i have large files logged no problems when log is done. Only partial downloads not logged
Because of speed, i don't need large two JS scripts used by J1.5, language UTF-8, i need 8859-1. I use sprites, some few tables with sprites. Most of components and static content come from one domain i have without cookies, remository is the only component i don't change to much. The goal is speed, i haven't problems even with 15000 user daily and large download bandwith.
One of many download managers net can do this
Download Accelerator Plus 9.4.1.1->http://download.cnet.com/Download-Accelerator-Plus/3000-2071_4-10037157.html
Please check it.
Thanks
Post edited by: bitpt, at: 2010/08/10 03:17
Update:
is not dificult log partial content, something like this
# $download_bytes_position = $start_point;
# while(!feof($open_download_file) && $download_bytes_position <= $end_point) {
# //mark download as aborted if connection has been closed
# if(connection_aborted() || connection_status() != 0)
# throw new Exception('Connection aborted');
#
# //calculate next chunk size
# $download_data_chunk_size = 1048000; (in this case 65536)
# if($download_bytes_position + $download_data_chunk_size > $end_point + 1)
# $download_data_chunk_size = $end_point - $download_bytes_position + 1;
#
# //get data chunk
# $download_data_chunk = fread($open_download_file, $download_data_chunk_size);
# if(!$download_data_chunk)
# throw new Exception('Could not read file');
#
# //output it
# // print($download_data_chunk);
# // flush();
#
# //increment download point
# $download_bytes_position += $download_data_chunk_size;
# }
Let me see if i can put this work in download_controller
Post edited by: bitpt, at: 2010/08/10 17:47
|
|
|
| | Sorry, you do not currently have permission to write here. |
bitpt
Karma: 0
|
Re:Multi Downloads Issue - 2010/08/11 04:29
I find a archaic solution for now.
all resumed downloads aren't logged, in resumed downloads
$_SERVER['HTTP_RANGE'] isn't empty.
in remository_download_Controller.php i change
if (!$fileinfo->islocal) {
/* remove logging here */
// $this->logDownload ($id, $user->id, $database, 0);
$slimurl = trim($fileinfo->url);
header("Location:$slimurl");
exit;
}
$ctype = $this->setCtype($file_extension, $downpath);
$offset = $origoffset = $this->rangeHandler($len);
/* logging here */
if (empty($_SERVER['HTTP_RANGE'])) {
$this->logDownload ($id, $user->id, $database, $len);
/* this is for new versions */
// $this->logDownload ($id, $user, $database, $len, 0);
}
This solve and stop for now multi download content without need stop resumed downloads.
If resume don't work, no problems at all, because log is done for each download.
This solution insert two lines in download_log, i increase limit two times for now.
In next days, i hope, find a better and clean solution to log all downloads.
Post edited by: bitpt, at: 2010/08/11 12:50
|
|
|
| | Sorry, you do not currently have permission to write here. |
|
Forum
