LinHES Forums
http://forum.linhes.org/

Robots.txt how to not let your mythbox get indexed
http://forum.linhes.org/viewtopic.php?f=5&t=17076		 Page 1 of 1
Author:  mattbatt [ Wed Oct 24, 2007 12:30 am ]
Post subject:  Robots.txt how to not let your mythbox get indexed
All the legit search engines abide by a w3 standard detailed here http://www.robotstxt.org/wc/robots.html
Put simply you ask them not to index your site or a portion of your site and they wont.
I don't want my mythtv indexed by Google, Yahoo, Alta Vista, Lycos, Web Crawler, and the like, so I put a "robots.txt" file in the /var/www/ folder. The text file simply states:
Code:
User-agent: *
Disallow: /

That means
"*" all robots
"/" don't look anywhere
I had to find the folder that the Knoppmyth default page was in so I thought I would document it here. Ya know kinda give back to the community as it were.

Thanks to all those who helped me.
MattBatt
Author:  Dale [ Wed Oct 24, 2007 1:36 am ]
Post subject: 
Your site should NOT be accessible from the internet. It is NOT safe, I don't care how much you may think that it is.
Author:  graysky [ Wed Oct 24, 2007 1:51 am ]
Post subject: 
Agreed... hackers would much rather get into a LINUX box than a windows box, so if you're mythtv box isn't behind a hardware firewall/router, you should have your head examined as it is likely an attractive target to people who know what they're doing. Further, do NOT forward any of the ports to your boxes -- particularly port 80!
Author:  rando [ Wed Oct 24, 2007 9:08 am ]
Post subject: 
If you are behind a firewall/router, but have port 80 (or some other port) forwarding your web traffic, would putting a robots file not be a good idea? As at that point your box isn't live on the internet, but yet is still accessible to a crawler, no?
Author:  Gibble [ Wed Oct 24, 2007 9:21 am ]
Post subject: 
Anybody can ignore robots.txt...it's like a "no trespassing" sign, it only keeps the honest folk out.
Author:  Girkers [ Wed Oct 24, 2007 3:43 pm ]
Post subject: 
rando, by forwarding port 80 to your KnoppMyth box you are making your box "live" on the internet, just like any other web server.
Author:  rando [ Wed Oct 24, 2007 4:26 pm ]
Post subject: 
When I said "live" I meant a box that is directly connected to the internet. Having a single port forwarded will be more secure than having a box connected directly and having all services/ports available to be accessed via the internet.

Perhaps the security risks are still pretty big with port 80 forwarded... but certainly they are less big than if all ports were accessible.
Author:  mattbatt [ Wed Oct 24, 2007 7:48 pm ]
Post subject: 
My mythweb page is being port forwarded from a non standard port. For example port 123456 is being port forwarded to port 80 on my mythbox. No that does not make it absolutely hacker proof but it helps. I also followed http://knoppmythwiki.org/index.php?page ... thWebHowTo
and changed my passwords and such so that I'm still less at risk.

One of the main benefits of Mythtv is the ability to change recording options from work. (It also is really cool to show Mac Fanboys why linux rocks.) So closing it off from the web is not a happy option.

Please tell me if I'm still insecure I don't want my baby hacked.
Author:  tjc [ Wed Oct 24, 2007 8:29 pm ]
Post subject: 
Of course it's still insecure, haven't people ever heard of port scanners? :? Add a bit of traffic sniffing to target the ports being probed and it can have the bad guys knocking on the right port with the right protocol within seconds. Oh, and the botnets running on at least half your neighbors MS-Windows boxes give them scores of machines in your network neighborhood to do the sniffing and probing...

If you expose any ports at all they should either be encrypted SSL type stuff, preferably with originating address limits, or tied to really bombproof servers isolated in a DMZ.
Author:  fra [ Wed Oct 24, 2007 10:06 pm ]
Post subject: 
tjc wrote:
Of course it's still insecure, haven't people ever heard of port scanners? :? Add a bit of traffic sniffing to target the ports being probed and it can have the bad guys knocking on the right port with the right protocol within seconds. Oh, and the botnets running on at least half your neighbors MS-Windows boxes give them scores of machines in your network neighborhood to do the sniffing and probing...

If you expose any ports at all they should either be encrypted SSL type stuff, preferably with originating address limits, or tied to really bombproof servers isolated in a DMZ.


I second that. I only have a port open for ssh. I then use ssh tunneling to access mythweb and manage the recordings. Moreover, I setup ssh to use a key not a password, just in case I also setup https for mythweb, and I only allow access from my work ip address.

Here's what I've done:

Edited the file /etc/ssh/sshd_config (as root) and made the following changes:
Code:
Port [choose something other than 22]
PermitRootLogin no
ChallengeResponseAuthentication yes
UsePAM no
ClientAliveInterval 60
ClientAliveCountMax 30

The last two lines are not in the file. I recommend adding them.
To figure out what these mean, type "man sshd_config".

If then on your router you forward the port you chose above onto your box, you can access it remotely:
Code:
ssh [your user name on the box]@[your ip address] -p [the port number]


To figure out the ip address of any machine you can use ifconfig or simply open a web browser onto www.whatismyipaddress.com

Once ssh is correctly setup, you can run remotely
Code:
ssh -L 8080:localhost:80 [your user name]@[your ip address] -p [the port number]

and then on your browser go to http://localhost.8080/mythweb
your web session will be tunneled through your ssh session and securely encrypted.

you can change the number 8080 to any number you want. just have to do it in the ssh command and in the browser. more importantly, you can change port 80 to a different number. but then you have to specify that port to apache on your box. to do this simply edit /etc/apache2/ports.conf and change the port number therein.

using this method, you haven't opened a new port! and the ssh port (which is the ONLY port you have open) is fairly secured, since it requires a password to get through.

of course, requiring key authentication rather than password authentication, restricting access to only a few chosen ip addresses, and setting up SSL (to use https) for mythweb, would make the whole business even more secure. all this exists in the wiki. i'm also in the process of writing my own wiki that integrates all that i've done to my box! it's coming along quite well and i'm hoping to post it soon. it'll include all the above info as well (about key authentication and the such).
Author:  Girkers [ Thu Oct 25, 2007 6:31 am ]
Post subject: 
Another option is VPN which is the method I use. I can't really comment on the security of this solution, it works and has the appearance of security.
Author:  rando [ Thu Oct 25, 2007 8:15 am ]
Post subject: 
Fra, I really like the sounds of your setup. I might do something similar. I really do like having access from "out and about" It's handy to check recordings, check my graphs, and just generally take a peek at the system to make sure everything is behaving.

Thanks for posting details of this (even though it sounds like they were/are already posted elsewhere).
Author:  rando [ Thu Oct 25, 2007 8:32 am ]
Post subject: 
This got me looking around.... and I think I've come across a document that appears to basically be doing what you've described.... I've only glanced over it and haven't given it a full proper read/try yet.

http://souptonuts.sourceforge.net/sshtips.htm

If I get some time today I'll be giving this a try and will report back on what kinda results I get.
Author:  rando [ Thu Oct 25, 2007 9:04 am ]
Post subject: 
Well, I've closed port 80 on my router, and followed the very easy steps in that URL to setup putty on my WinXP machine to access my MythTV system and it works like a charm. With a simple change to my putty config I'm good to go. As long as I'm SSHd into my box I can browse to http://localhost and have that traffic redirected (securely!) to my mythweb page.
Author:  mac [ Thu Oct 25, 2007 10:18 am ]
Post subject: 
iptables is not rocket science. Not sure why people put so more faith in a closed source firewall than one they have complete control over. iptables is as bulletproof or more than any appliance and in many cases, it is what an appliance is using.

If you have a fairly simple set of rules on the myth box there is no reason that it can't be connected directly to the internet.. For example:

# Allow loopback connections
iptables -A INPUT -p TCP -s 127.0.0.1 -j ACCEPT
# Allow everything from my non-routable IP address
iptables -A INPUT -p TCP -s 10.111.111.111 -j ACCEPT
# Allow the box to get out and do almost anything TCP related is wishes, eth0 faces the outside.
iptables -A INPUT -i eth0 -p TCP -m state --state ESTABLISHED,RELATED -j ACCEPT
# Ignore all TCP not related to connections initiated by me.
iptables -A INPUT -p TCP -j DROP
# Take care of nameservice.
iptables -A INPUT -p UDP -s <my nameserver> --source-port 53 -j ACCEPT
iptables -A INPUT -p UDP -j REJECT

Doing something like forwarding port 80 is not a big deal in that you can restrict the rule to just one incomming IP address. You can even turn the rule on an off as you need it.

etc.. you should run nmap against your own box to make sure you did things correctly.
---- just quick SYN scan from an outside ip address to see any holes :
foo>nmap -sS x.x.x.x
Starting nmap 3.45 ( http://www.insecure.org/nmap/ ) at 2007-10-25 22:38 KGST
All 1657 scanned ports on x.x.x.x are: filtered
Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/