Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today!

If you have any problems with the registration process or your account login, please contact us.

Post New Thread Reply

Register GFY Rules Calendar Mark Forums Read
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >
New Webmasters ask "How-To" questions here. This is where other fucking Webmasters help.

 
Thread Tools
Old 10-06-2017, 03:25 PM   #151
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
no in bash.sh the ; at the end of a statement is not needed
var=something (declaration)
like JavaScript
$var beneath (a declared variable)
like
echo $var

the caps are what i did they could be in lowercase too -- but bash .sh is case sensitive

in a terminal

$ dothis; sothat; is this; && dosomethindgood | (<pipe>) to the next statement
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-09-2017, 01:19 PM   #152
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
So,
I did that...
Code:
#!/bin/bash
# Shell script to backup MySql database
# To backup Nysql databases file to /backup dir and later pick up by your
# script. You can skip few databases from backup too.
# For more info please see (Installation info):
# http://www.cyberciti.biz/nixcraft/vivek/blogger/2005/01/mysql-backup-script.html
# Last updated: Aug - 2005
# --------------------------------------------------------------------
# This is a free shell script under GNU GPL version 2.0 or above
# Copyright (C) 2004, 2005 nixCraft project
# -------------------------------------------------------------------------
# This script is part of nixCraft shell script collection (NSSC)
# Visit http://bash.cyberciti.biz/ for more information.
# -------------------------------------------------------------------------
STARTTIME=date +%s

MyUSER=root     # USERNAME
MyPASS=Alfarenna79   # PASSWORD
MyHOST=localhost        # Hostname
 
# Linux bin paths, change this if it can't be autodetected via which command
MYSQL="$(which mysql)"
MYSQLDUMP="$(which mysqldump)"
CHOWN="$(which chown)"
CHMOD="$(which chmod)"
GZIP="$(which gzip)"
 
# Backup Dest directory, change this if you have someother location
DEST="/var/backup"
 
# Main directory where backup will be stored
MBD="$DEST/mysql"

#elimino vecchi backup
rm $MBD/*
 
# Get hostname
HOST="$(hostname)"
 
# Get data in dd-mm-yyyy format
NOW="$(date +"%d-%m-%Y")"
 
# File to store current backup file
FILE=""
# Store list of databases
DBS=""
 
# DO NOT BACKUP these databases
IGGY="information_schema cond_instances mysql performance_schema phpmyadmin"
 
[ ! -d $MBD ] && mkdir -p $MBD || :
 
# Only root can access it!
$CHOWN 0.0 -R $DEST
$CHMOD 0600 $DEST
 
# Get all database list first
DBS="$($MYSQL -u $MyUSER -h $MyHOST -p$MyPASS -Bse 'show databases')"
 
for db in $DBS
do
    skipdb=-1
    if [ "$IGGY" != "" ];
    then
        for i in $IGGY
        do
            [ "$db" == "$i" ] && skipdb=1 || :
        done
    fi
 
    if [ "$skipdb" == "-1" ] ; then
        #FILE="$MBD/$db.$HOST.$NOW.gz"
        #no gzip, comprimo dopo tutta la cartella
        FILE="$MBD/$db.$HOST.$NOW.sql"

        # do all inone job in pipe,
        # connect to mysql using mysqldump for select mysql database
        # and pipe it out to gz file in backup dir :)
        #$MYSQLDUMP -u $MyUSER -h $MyHOST -p$MyPASS $db | $GZIP -9 > $FILE

        #no gzip, comprimo dopo tutta la cartella
        $MYSQLDUMP -u $MyUSER -h $MyHOST -p$MyPASS $db > $FILE
    fi
done

#comprimo tutto
zip -r $DEST/mysql-backup-$HOST.zip $MBD/

#tar -zcvf $DEST/mysql-backup-$HOST.tar.gz $MBD

ENDTIME=date +%s

TOTTIME=$ENDTIME-$STARTTIME

echo Elapsed_time: $TOTTIME
But it tells me
Code:
/var/backup/mysql_backup: line 15: +%s: command not found
It seems that he does not like the +%s, but I really do not know how to solve it...
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-10-2017, 05:03 AM   #153
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
STARTTIME=(`date +%s`)

try like this and the time will be in epoch seconds

ENDTIME=(`date +%s`)
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-10-2017, 02:52 PM   #154
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
It works, but the result is kinda odd...

Code:
Elapsed_time: 1507671766-1507671705
Practically he tells me: the calculation made you
I tried to put quotes, parentheses etc etc, but he does not want to do it... can we do this last thing too?

It takes more time for this little thing than to configure all the server...
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-10-2017, 10:15 PM   #155
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
Maybe in $()
TOTTIME=$($ENDTIME-$STARTTIME)

Do the math -- the sum is in seconds

[email protected]:~$ bc <<< 1507671766-1507671705
61
seconds
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-11-2017, 02:30 PM   #156
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
Says:
Code:
/var/backup/mysql_backup: line 93: 1507756208-1507756139: command not found
But looking for "bc" (which I did not know), I found this:
Code:
TOTTIME=`expr $ENDTIME - $STARTTIME`
This seems to work

Now I'm worried about those odd quotes...
In PHP when I find those quotes it means that there was a copy paste error from the HTML and nothing works anymore.

So I have the habit of removing them as soon as I see them and replace them with a normal apex... in sh instead it seems to be fundamental... I surely have removed someone thinking they were a error...

I shouldn't have done any damage, because everything seems to work, but maybe I'm going to look for the original script and I see if there was someone...

P.S. It's strange how we can install an entire server, and then the simplest things make us crazy...
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-11-2017, 03:03 PM   #157
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
these are called backticks

bc is a terminal calculator program

apt install bc

man bc

Quote:
The backtick (``) is actually called command substitution. The purpose of command substitution is to evaluate the command which is placed inside the backtick and provide its result as an argument to the actual command. The command substitution can be done in two ways one is using $() and the other is "``" .Dec 22, 2011
look above^^

Quote:
STARTTIME=(`date +%s`)

try like this and the time will be in epoch seconds

ENDTIME=(`date +%s`)
I just habitually close an expression in () for clarity in my code -- it is probably in most cases superfluous (but benign)
Like
our @array =(<FILENAME>);
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-12-2017, 02:34 PM   #158
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
There's one last thing that scares me a lot

Load


Much of that red is due to the phase of moving sites and all the importing error of those damned databases.

Also the other server at the beginning had very red, then slowly it is normalized.

This is taking a little more...

But what sounds strange to me is that going to see the detail of the server, you do not understand why there is all that red.



The CPU rarely arrives at 90%, the memory is a bit chubby but it works, disk there is plenty, errors or special problems there are none...
The sites are running well, fast, without interruptions, or visible slowdowns...

Cpu sometimes says "stolen" even if it is working maybe at 70%, and already this is odd.

But it is the usual load to give more worries, sometimes even 4-5, I also saw 7 in the days of cronjob (they are still synchronizing many data due to the lack of cronjob in the other servers)

What does it actually indicate load?

And how much do I have to worry?

On a scale it goes from "quiet, goes all right" to "shit the server is going to explode, run away all before it's late, shit we'll die all ", where am I?
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-12-2017, 06:58 PM   #159
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
Your problem is your PHP script and the MySQL daemon (server). Software for your application; or,
If you look at the times of the peak usage and grep those times in the server access logs you may find that bing is indexing too many pages too fast -- you can place a directive in the robots.txt
User-agent: bingbot
Crawl-delay:$v
5
10

see
https://www.siteground.com/kb/how_to..._eng ine_bot/
https://www.bing.com/webmaster/help/...ntrol-55a30302

Slow bing down -- don't Disallow Bing they bring good converting traffic and the sell their PSaaS or indexed database to Yahoo and other search engines.

You may find Baidu is indexing too many pages too fast -- block them at your firewall I have had luck that way
Porn is illegal in China and you won't sell legit Chinese buyers either.
# Free IP2Location Firewall List by Search Engine
# Source: Whitelist Robots by Search Engine | IP2Location

Code:
whois -h v4.whois.cymru.com " -c -p 183.131.32.0/20"
AS      | IP               | BGP Prefix          | CC | AS Name
4134    | 183.131.32.0     | 183.128.0.0/11      | CN | CHINANET-BACKBONE No.31,Jin-rong Street, CN
 -c -p 12.1.72.32/27"
7018    | 12.1.72.32       | 12.0.0.0/9          | US | ATT-INTERNET4 - AT&T Services, Inc., US
" -c -p 104.193.88.0/22"
55967   | 104.193.88.0     | 104.193.88.0/24     | US | CNNIC-BAIDU-AP Beijing Baidu Netcom Science and Technology Co., Ltd., CN
or use Ruby gem nicinfo
https://github.com/arineng/nicinfo

that will give you full RDAP/whois information

Third way is just $ whois <ip address>

If you are generating many dynamic pages search engines may be causing this problem

Scrapers and *bad bots* may be the issue too.

This is what server logs are for to search for problems and find patterns.
A firewall is the way to go -- just do not answer -- drop the packet.
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-16-2017, 03:40 PM   #160
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
But Holy cow

I was away 2 days and the server was invaded by bots, just like you said...



Code:
51.255.65.66 - - [16/Oct/2017:22:25:31 +0000] "GET /27 HTTP/1.1" 302 3634 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
157.55.39.234 - - [16/Oct/2017:22:25:09 +0000] "GET /cimla+sexy+photos.com/ HTTP/1.1" 200 32929 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
93.105.187.11 - - [16/Oct/2017:22:25:05 +0000] "GET /search.php?q=shemale+mia+isabella+teacher+her+student+a+lesson+free+porn&sort=date&page=5 HTTP/1.1" 200 10438 "http://www.bigbigbigboobs.com/search.php?q=shemale+mia+isabella+teacher+her+student+a+lesson+free+porn&sort=date" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
216.244.66.245 - - [16/Oct/2017:22:25:30 +0000] "GET /search-amy+anderssen+photos+pk/ HTTP/1.1" 200 79767 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
207.46.13.86 - - [16/Oct/2017:22:25:24 +0000] "GET /search-bigboob+s+saree+woman+photo+pk/ HTTP/1.1" 200 24426 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
::1 - - [16/Oct/2017:22:25:31 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
66.249.64.3 - - [16/Oct/2017:22:25:16 +0000] "GET /love+sex+move/ HTTP/1.1" 200 34386 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
40.77.167.14 - - [16/Oct/2017:22:25:31 +0000] "GET /search-bbw+back+sid+girl+photos.com/ HTTP/1.1" 200 24797 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
66.249.70.19 - - [16/Oct/2017:22:25:32 +0000] "GET /74277 HTTP/1.1" 200 19774 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.30 - - [16/Oct/2017:22:25:32 +0000] "GET /savita+babhi/ HTTP/1.1" 200 20327 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
[email protected]:~# tail /var/log/apache2/access.log
8.37.233.40 - - [16/Oct/2017:22:26:54 +0000] "GET /download+video+bokep+jepang+rina+araki/ HTTP/1.1" 200 29346 "https://www.google.co.id/search?client=ucweb-b-bookmark&q=video+ngentot+rina+araki&oq=video+ngentot+rina+araki&aqs=mobile-gws-lite.." "Mozilla/5.0 (Linux; U; Android 6.0.1; en-US; SM-G532G Build/MMB29T) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 UCBrowser/11.3.5.972 U3/0.8.0 Mobile Safari/534.30"
49.34.127.70 - - [16/Oct/2017:22:26:56 +0000] "GET /xvillage+desi+8+saal+ki+bachi+ki+chudai+video/ HTTP/1.1" 200 31479 "android-app://com.google.android.googlequicksearchbox" "Mozilla/5.0 (Linux; Android 5.1.1; SM-J200G Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36"
93.105.187.11 - - [16/Oct/2017:22:26:30 +0000] "GET /page-17/search-googleweblight.comlite_url+2+mom+big+naked+milky+boobs+images.com/date/ HTTP/1.1" 200 26231 "http://www.monsterboobshardpics.com/page-14/search-googleweblight.comlite_url+2+mom+big+naked+milky+boobs+images.com/date/" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
216.244.66.228 - - [16/Oct/2017:22:27:08 +0000] "GET /search-big+assas+larag+ass+masive+ass+huge+cock+large+cock+hardcore+anal+gp+download+free/ HTTP/1.1" 200 97124 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
93.105.187.11 - - [16/Oct/2017:22:26:18 +0000] "GET /page-7/search-desi+bhabi+sexy+boob+press+fuck+pussy+mp+mobile+ipone/date/ HTTP/1.1" 200 24021 "http://www.monsterboobshardpics.com/search-desi+bhabi+sexy+boob+press+fuck+pussy+mp+mobile+ipone/date/" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
93.105.187.11 - - [16/Oct/2017:22:26:23 +0000] "GET /page-14/search-boobs+milk+breathing+imeges/date/ HTTP/1.1" 200 24406 "http://www.monsterboobshardpics.com/page-9/search-boobs+milk+breathing+imeges/date/" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
207.46.13.183 - - [16/Oct/2017:22:27:07 +0000] "GET /page-15/search-african+black+pussy/ HTTP/1.1" 200 24382 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
202.46.58.190 - - [16/Oct/2017:22:27:06 +0000] "GET /search-big+black+fatty+boom+shemale+fuck/ HTTP/1.1" 200 24940 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
51.255.65.27 - - [16/Oct/2017:22:27:09 +0000] "GET /36331 HTTP/1.1" 200 19518 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
93.105.187.11 - - [16/Oct/2017:22:26:08 +0000] "GET /page-12/search-tite+big+round+heavy+boobs+hd+pics/date/ HTTP/1.1" 200 25087 "http://www.monsterboobshardpics.com/page-7/search-tite+big+round+heavy+boobs+hd+pics/date/" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
[email protected]:~# tail /var/log/apache2/access.log
216.244.66.228 - - [16/Oct/2017:22:27:24 +0000] "GET /search-bbw+big+hips+mom+churidar+hot+photo+xxxin/ HTTP/1.1" 200 92852 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
46.229.168.79 - - [16/Oct/2017:22:27:25 +0000] "GET /52199 HTTP/1.1" 200 19569 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
93.105.187.11 - - [16/Oct/2017:22:27:06 +0000] "GET /search.php?q=leanne+crow+huge+boobs+fake&page=5 HTTP/1.1" 200 10927 "http://www.bigbigbigboobs.com/search.php?q=leanne+crow+huge+boobs+fake" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
93.105.187.11 - - [16/Oct/2017:22:27:02 +0000] "GET /search.php?q=windian+bhabi+tight+salwar+gand+penty+showing+sexy+pic&page=2 HTTP/1.1" 200 10397 "http://www.bigbigbigboobs.com/search.php?q=windian+bhabi+tight+salwar+gand+penty+showing+sexy+pic" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
::1 - - [16/Oct/2017:22:27:26 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
157.55.39.77 - - [16/Oct/2017:22:27:24 +0000] "GET /page-13/search-pornstar+aunty+sex+videos+downloadiporntv.net/ HTTP/1.1" 200 26726 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
216.244.66.233 - - [16/Oct/2017:22:27:24 +0000] "GET /303/ HTTP/1.1" 200 79410 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])"
93.105.187.11 - - [16/Oct/2017:22:27:16 +0000] "GET /?q=face+book+hot+nice+aunty+xxx+back+side+imagedate/ HTTP/1.1" 200 9673 "http://www.bigbigbigboobs.com/search.php?q=face+book+hot+nice+aunty+xxx+back+side+imagedate&page=6" "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
40.77.167.62 - - [16/Oct/2017:22:27:25 +0000] "GET /search-anteysex+photo.com/ HTTP/1.1" 200 23531 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
180.76.15.8 - - [16/Oct/2017:22:27:26 +0000] "GET /page-3/search-japanese+boobs+pics/random/ HTTP/1.1" 500 637 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
[email protected]:~# tail /var/log/apache2/access.log
157.55.39.238 - - [16/Oct/2017:22:27:20 +0000] "GET /page-5/search-african+aunty+without+dress+and+bra+big+boobs+sexy+photos/ HTTP/1.1" 200 24875 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.20 - - [16/Oct/2017:22:27:28 +0000] "GET /page-16/search-sa+tranny+nude+pics/ HTTP/1.1" 200 25587 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.39 - - [16/Oct/2017:22:27:21 +0000] "GET /desi+girl+in+loose+tshirt+pics/ HTTP/1.1" 200 27165 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.39.29 - - [16/Oct/2017:22:27:26 +0000] "GET /page-14/search-hot+sexy+aunty+boobs+in+saree+hd+picturescom/ HTTP/1.1" 200 25844 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
202.46.57.88 - - [16/Oct/2017:22:27:28 +0000] "GET /page-5/search-naked+pics+of+nicole+charming/ HTTP/1.1" 200 24457 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
157.55.39.149 - - [16/Oct/2017:22:26:51 +0000] "GET /page-2/search-big+boobs+pandora+peaks+bikini/ HTTP/1.1" 200 0 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
202.46.58.166 - - [16/Oct/2017:22:27:28 +0000] "GET /search-lesbian+sucking+boobs/random/ HTTP/1.1" 200 24133 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
164.132.161.3 - - [16/Oct/2017:22:27:31 +0000] "GET /7241 HTTP/1.1" 302 3638 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
207.46.13.152 - - [16/Oct/2017:22:27:30 +0000] "GET /search-big+boobs+tite+studant/ HTTP/1.1" 200 23030 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
46.229.168.67 - - [16/Oct/2017:22:27:27 +0000] "GET /sunnyleone%20sexbeg/ HTTP/1.1" 200 20184 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
Now I see to read and understand quickly everything you wrote me

Thanks, just in time
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-16-2017, 06:11 PM   #161
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
So, I limited Bing from robots.txt on all my sites. For now I see no big differences, but maybe it takes a little because of the cache

I also found in my htaccess, these rules that should stop Yandex and China

Code:
RewriteCond %{HTTP_USER_AGENT} ^.*MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Yandex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Baidu [NC]
RewriteRule .*  - [L,F]

Then I went to see Ip2location, I joined and I generated the file, but I did not understand how to use the file that they gave me...

I have generated Linux iptables, they gave me a thing like this:

Code:
iptables -A INPUT -s 104.146.0.0/18 -j DROP
iptables -A INPUT -s 104.146.100.0/22 -j DROP
iptables -A INPUT -s 104.146.104.0/21 -j DROP
iptables -A INPUT -s 104.146.112.0/24 -j DROP
But on Ubuntu in Digitalocean there is UFW, how can I use the file ip2location in UFW

Do I need to install iptables? Will it still work UFW?

I have seen some sites where it says to open a configuration file of UFW and add the lines, but my lines have a different format and the files to be modified indicated in these sites are always different...

I also thought of changing manually this:
Code:
iptables -A INPUT -s 104.146.100.0/22 -j DROP
in this:
Code:
# block IP
-A ufw-before-input -s 104.146.100.0/22 -j DROP
and add the lines to the/etc/ufw/before.rules file, as shown in Ubuntu Wiki

But I'm not sure that doing this manually is a good idea

I'm not really understanding anything.

And I would not use the rules in httacces, first because here also have a different format from what I used in precedence, and then because they are so many...

Do I need to install Fail2ban?

I really need to fix this thing quickly because my server is merging, can you help me?
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-17-2017, 06:09 AM   #162
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
Code:
ufw deny from 192.187.100.58 to any;
ufw deny from 112.137.167.30 to any;
ufw deny from 82.117.194.229 to any;
ufw deny from 91.121.45.246   to any;
deny an ip in UFW

Code:
[email protected]:/home/work# ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22                         ALLOW IN    99.30.xxx.xx/29            
[ 2] 80                         ALLOW IN    Anywhere                  
[ 3] 443                        ALLOW IN    Anywhere                  
[ 4] xxxx                       ALLOW IN    99.30.xxx.xx/29            
[ 5] 80,443/tcp                 ALLOW IN    Anywhere                  
[ 6] Nginx Full                 ALLOW IN    Anywhere                  
[ 7] Anywhere                   DENY IN     69.30.222.130             
[ 8] Anywhere                   DENY IN     155.133.82.122            
[ 9] Anywhere                   DENY IN     54.196.30.74              
[10] Anywhere                   DENY IN     66.240.205.0/26           
[11] Anywhere                   DENY IN     188.165.2.183             
[12] Anywhere                   DENY IN     71.6.146.130              
[13] Anywhere                   DENY IN     89.163.146.57             
[14] Anywhere                   DENY IN     139.162.199.176           
[15] Anywhere                   DENY IN     180.97.106.37             
[16] Anywhere                   DENY IN     104.193.252.165           
[17] Anywhere                   DENY IN     190.248.153.234           
[18] Anywhere                   DENY IN     142.54.183.226            
[19] Anywhere                   DENY IN     158.106.67.0/24           
[20] Anywhere                   DENY IN     170.210.156.91            
[21] Anywhere                   DENY IN     81.4.125.125              
[22] Anywhere                   DENY IN     66.240.192.128/26         
[23] Anywhere                   DENY IN     35.188.194.96             
[24] Anywhere                   DENY IN     149.202.207.121           
[25] Anywhere                   DENY IN     158.106.64.0/18           
[26] Anywhere                   DENY IN     142.54.161.10             
[27] Anywhere                   ALLOW IN    99.30.xx.xx/29 21         
[28] Anywhere                   DENY IN     66.240.192.0/18           
[29] Anywhere                   DENY IN     192.187.100.58            
[30] Anywhere                   DENY IN     112.137.167.30            
[31] Anywhere                   DENY IN     82.117.194.229            
[32] Anywhere                   DENY IN     91.121.45.246             
[33] 80 (v6)                    ALLOW IN    Anywhere (v6)             
[34] 443 (v6)                   ALLOW IN    Anywhere (v6)             
[35] 80,443/tcp (v6)            ALLOW IN    Anywhere (v6)             
[36] Nginx Full (v6)            ALLOW IN    Anywhere (v6)
Code:
[email protected]:/home/work# ufw delete 37
Deleting:
 allow 21/tcp
Proceed with operation (y|n)? y
Rule deleted
you have to run ufw status numbered each time you delete as the numbered order will change if you delete any number other than the last (end) number -- the rule chain will change.

Mapping the rules is a better idea but I haven't seen a good solution for ufw only for iptables and now nftables

ufw is an acronym for Uncomplicated FireWall
UFW: The Linux Uncomplicated Firewall <uncomplicated tutorial

iptables is sort of hard to understand and has been superseded by https://linux-audit.com/nftables-beg...fic-filtering/ <nftables

Baidu doesn't play by the rules regarding robits.txt
and will use IP to spider you without any user-agent sig that says 'baidu' making you .htaccess code useless.

get the ip CIDRs and block them in the ufw firewall.
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-17-2017, 04:23 PM   #163
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
So, I downloaded the CIDR of the engines that I want to block



and launched this:
Code:
while read line; do sudo ufw insert 1 deny from $line to any; done < cdir.txt
And he have added about 650 new rules in UFW

But in the access.log the ones I see most often are:
Opensiteexplorer.org/dotbot, [email protected]
semrush.com/bot.html
bing.com/bingbot.htm
ahrefs.com/robot/

Apart from Bing, the rest seem to be marketing tools, some more or less connected to Google or moz.com

I don't use them, but mostly I don't need them if they first destroyed my server...

Can I block them? Always via IP-UFW?

And in case, which IP should I block?
Their Ip in my access.log changes, eg:

Code:
46.229.168.76 - - [17/Oct/2017:23:04:53 +0000] "GET /search-busty%20mom%20loves%20to%20suck%20cock/ HTTP/1.1" 200 24789 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
46.229.168.69 - - [17/Oct/2017:22:57:38 +0000] "GET /search-big%20brest%20sex%20photo/random/ HTTP/1.1" 200 21170 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
46.229.168.67 - - [17/Oct/2017:22:55:13 +0000] "GET /desi%20girls%20boobs%20suckers%20bees%20photos2015/ HTTP/1.1" 200 31832 "-" "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"
How do I find all their IP?

Or is it better in this case to use robots.txt? Is there a serious list (robots.txt or IP) of bad bots to block?
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-18-2017, 05:00 PM   #164
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
So, in the meantime... I don't know if I made a crap...

But I did this thing...

I searched with grep some bots in the acces.log
Code:
grep ahrefs/var/log/apache2/access.log

Then I copied a few thousand lines and I created a PHP script that creates a file with only the IP, line by line (leaving the duplicates)
Code:
<?
$my_database_txt = 'seznambot.txt';

$array_righi = file($my_database_txt);
foreach($array_righi as $key => $capi){
    list($ip, $merda) = explode(" - - ", $capi);

    if(strpos($ip_list, $ip) === false){

        $ip_list = $ip_list.$ip.'
';

    }
}


$nome_file_index = "ip_list.txt";
//chmod($nome_file_index, 0666);
$file = fopen($nome_file_index, "w") ;
	fwrite($file, $ip_list);
	fclose($file);
?>
From all the games for now came out this file
http://porn-update.com/temp/bad-bot-cidr.txt

Then with the usual while I added the rules to UFW
Code:
while read line; do sudo ufw insert 1 deny from $line to any; done < /var/www/html/bad-bot-cidr.txt
If I did some crap I should be able to go back with this:
Code:
while read line; do sudo ufw delete deny from $line; done < /var/www/html/bad-bot-cidr.txt
Now it seems that something is happening, in the access.log I see a lot of Google and Bing, although for Bing I added crawl-delay: 1 in robots.txt and I also set to 1 the scan frequency on Bing webmaster tools for all My sites.

The CPU graph goes up and down at the moment, but that of load is slowly descending.



I wait a little, and I see what happens...
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-18-2017, 06:47 PM   #165
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
If the only tool in your toolbox is a hammer that is how you screw in a wood screw ...

try this:

Code:
$ cut -d'-' -f1 /home/work/domain.com/logs/access.log| grep -v '99\.3' |  uniq -c | sort -nr|sed 's/\([0-9]\) \([0-9]\)/\1:\2/g' |less

(returns unique hits:IP grep -v will delete your ip pattern)

   2742:173.208.249.226 
    189:158.69.229.6 
    155:160.202.163.148 
    153:78.190.44.124 
     91:82.165.75.132 
     64:178.137.82.201 
     62:46.2.77.72 
     62:201.18.18.173 
     62:201.18.18.173 
     62:185.81.155.40
to print to a file
instead of |less
>fileName.*

you don't need a hammer to tighten a screw

After checking the whois
Quote:
# The following results may also be obtained via:
# https://whois.arin.net/rest/nets;q=1...se&ext=netref2
#


# start

NetRange: 173.208.249.224 - 173.208.249.231
CIDR: 173.208.249.224/29
NetName: DS-249-225-231
NetHandle: NET-173-208-249-224-1
Parent: WII-OAK-2 (NET-173-208-128-0-1)
NetType: Reassigned
OriginAS: AS32097
Customer: Caruso, David (C06755517)
RegDate: 2017-10-18
Updated: 2017-10-18
Ref: https://whois.arin.net/rest/net/NET-173-208-249-224-1


CustName: Caruso, David
Address: 201 E. 16th st
City: North Kansas City
StateProv: MO
PostalCode: 64116
Country: US
RegDate: 2017-10-18
Updated: 2017-10-18
Ref: https://whois.arin.net/rest/customer/C06755517

OrgTechHandle: AWE13-ARIN
OrgTechName: Wendel, Aaron
OrgTechPhone: +1-816-256-3031
OrgTechEmail: [email protected]
OrgTechRef: https://whois.arin.net/rest/poc/AWE13-ARIN

OrgAbuseHandle: NETWO1111-ARIN
OrgAbuseName: Network Security
OrgAbusePhone: +1-816-256-3031
OrgAbuseEmail: [email protected]
Whoever the fuck he is ...

Code:
ufw deny from 173.208.249.224/29  to any;
Rule added
A CIDR is a group of IPs

__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-18-2017, 09:56 PM   #166
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
I thought to a fast way to get an IP CIDR

Code:
$ whois  173.208.249.226 |grep 'CIDR:'|cut -d':' -f2|sed -e 's/^/ufw deny from /g' -e 's/            / /g' -e 's/$/  to any;/g'

#returns
ufw deny from 173.208.249.224/29  to any;
ufw deny from 173.208.128.0/17  to any;
Use the smaller CIDR (narrower) of just the ip's if you dont want to block the whole datacenter (/18 to /24)

__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-19-2017, 06:04 PM   #167
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
So, something I did...

Now virtually are only the Google and Bing bots in my access.log.

But CPU and load are still at absurd levels...

The strangest thing, something has changed in both my servers....



This week the visits have not doubled (indeed, they have fallen a bit), but something is obviously changed, but I have no idea what it is...

I have not changed anything, I searched in all the logs I know, but I find nothing that can explain this sudden increase in CPU usage.

In the error logs I often find lines like this:
Code:
[Fri Oct 20 01:00:20.885615 2017] [core:error] [pid 4771] [client 37.9.113.202:36406] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
[Fri Oct 20 01:00:25.593807 2017] [core:error] [pid 4771] [client 37.9.118.28:37754] AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
But I have no idea what it relates to, my sites don't make any redirect, the only redirect I've ever created is in htaccess to redirect http -> https, or vicevers https -> http, but they seem to work properly.

These days I have been trying to look also often at the sites and seem to work well.

Nixstat often says that the mysqld process is often at 130%, 150%, 180%, my sites definitely make an important use of MySQL, but having not changed anything, I do not understand why.
(It was very high when I had problems importing tables, but it had normalized after having them fixed.)

Not having increased the visits, and having eliminated many of the bots, who or what is using my CPU and my mysql?

I don't know what else to do to understand what's going on...


I'm kinda worried also because usually the visits increase on Saturday and Sunday, and I have no idea what will happen this week
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-20-2017, 05:03 AM   #168
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
https://www.google.com/search?client...utf-8&oe=utf-8

https://gist.github.com/JustThomas/141ebe0764d43188d4f2

I usually try searching the exact error to get some idea -- seems this may be some .htaccess rewrite errors ...
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-20-2017, 08:02 PM   #169
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
t's the first thing I always do, I ask here when I find nothing

I found a lot of solutions for wordpres, but my sites are not WordPress, and I understand very little url_rewriter...

I wrote these rules a long time ago, following guides, and I never saw this error before, until I started to manage my server...

My url_revriter is really simple and stupid, and the error does not give many clues to understand what creates it.

Code:
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^.*MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Yandex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Baidu [NC]
RewriteRule .*  - [L,F]


RewriteRule ^page-([^/]+)/$ index.php?page=$1 [L]

RewriteRule ^([^/]+)\.html$ video.php?title=$1 [L]

RewriteRule channels/ channels.php [L]

RewriteRule tags/ tags.php [L]

RewriteRule ^tags-([^/]+)\/$ tags.php?letter=$1 [L]

RewriteRule search/ search.php [L]

RewriteRule ^search-([^/]+)$ search.php?query=$1 [L]
RewriteRule ^search-([^/]+)\/$ search.php?query=$1 [L]
RewriteRule ^page-([^/]+)/search-([^/]+)/$ /search.php?page=$1&query=$2 [L]

RewriteRule ^search-([^/]+)\/([^/]+)\/$ search.php?query=$1&sort=$2 [L]
RewriteRule ^page-([^/]+)/search-([^/]+)/([^/]+)/$ /search.php?page=$1&query=$2&sort=$3 [L]

RewriteRule ^([0-9]+)$ out.php?id_photo=$1 [L]
RewriteRule ^([0-9]+)/$ out.php?id_photo=$1 [L]
RewriteRule ^([0-9]+)-([^/]+)$ out.php?id_photo=$1 [L]

But I also found another thing... Last week Google decided to scan my sites, all together, and a lot of pages of each site...

Now I'm thinking, I don't know if...

Wait a few days, maybe, when Google has finished this scan the server returns to a normal regime.

Or

Try to limit its consumption by Google, adding for example if-midified-since and Last-Modified in the headers of my pages

There were already, but time ago I had to comment on why they created problems with some crappy VPS
porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-20-2017, 08:57 PM   #170
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
I had a problem with the new Apache2 version then changed to Nginx for other reasons.

Code:
	</IfModule>
	<Directory /home/xxxxx/xxx.xxx.com/public_html>
	   Order allow,deny
	   Allow from all
	   # New directive needed in Apache 2.4.3: 
	   Require all granted
	</Directory>
You might want to check for this in your configurations that it is there -- IDK if this is part of the issue or not.
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-21-2017, 05:11 PM   #171
porn-update
Confirmed User
 
porn-update's Avatar
 
Industry Role:
Join Date: Apr 2014
Posts: 324
But where?
I remember you had already shown me this thing, but for some strange reason in my .conf files there is not...

Where should I add, in the files mysitecom.conf?
In sites-enabled or sites-available? or both?

Then I have to disable and enable the sites?

Restart Apache clearly, but I have to reboot the server too?

I want to fix this, because my logs are still full of those errors,

Perhaps there is some new hope, in the last few hours something is changing, as it started, it seems perhaps to return to normal...

porn-update is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 10-21-2017, 06:42 PM   #172
Barry-xlovecam
It's 42
 
Barry-xlovecam's Avatar
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 17,570
Code:
cd /etc/apache2/sites-available
ls -1
cp file file.bk <making a backup copy
nano file
save
then
######
#then make symbolic link
Code:
a2ensite <file>
/etc/init.d/apache2 reload
######
or
service apache2 reload

reload as opposed to restart this just reloads the new configuration.
__________________
Vive la résistance -- Carpe diem

"If you only knew the power of the dark side." ...Darth Vader
Sk.why.pe: barry_555_5 -- ICQ: three1896three617
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Post New Thread Reply
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >

Bookmarks

Tags
vps, centos, cwp, thinking, line, free, command, software, advice, direction, cpanel, memcache, guides, correct, reverse, varnish, tool, litespeed, proxy, initial, acpu, opcache, apache, web, past
Thread Tools




©2000-, AVN Media Network Inc



Powered by vBulletin
Copyright © 2000- Jelsoft Enterprises Limited.