What Happens When You Request a Webpage

Okay, with all the securing all the things posts, it occurs to me that I haven’t actually explained this. (I did, however, explain this in depth for an interview at Large Internet Company, so it’s probably higher-level than I thought.) One of my first jobs included writing CGIs (web-based programs) that logged into and/or proxied other sites for our authenticated users. (That’s legit; it was a college that had paid for an online database service and wanted to provide access to their students. There are now products (google EZProxy, for example) that do this, but this was back in the wild wild west frontier. But I digress.)

Okay. You fire up your computer from a dead sleep. You launch a web browser and point it at http://www.google.com. What happens then?

The first thing that happens is your computer asks your DNS (Domain Name Server) for the IP address of www.google.com. Because these names are easy for humans to remember but computers still use the IP address. (You know what an IP address is, right? It’s a number that looks like–four numbers between 1 and 255 inclusive separated by dots. These numbers allow your computer to actually find the remote server on the internet. I’ll just leave that there for you to take as written unless someone really cares, LOL.) Unless you’re using something like Open DNS or Google’s DNS–basically, unless you change what your ISP gave you when they gave you your IP address–that DNS server is probably run by your ISP.

Your DNS server doesn’t actually store any IP addresses permanently (unless they’re Authoritative for a particular domain, like your ISP, and they might/probably actually separate those out between authoritative DNS servers and client query DNS servers). If someone else has requested www.google.com recently, however, it stores the answer for however long Google’s DNS servers told it it could cache the answer (this is called TTL, or Time To Live). If no one has requested www.google.com recently, or if the old answer has expired, it goes out to the servers that are authoritative for the .com domains and says, “Who’s authoritative for google.com?” Those domain servers give your local DNS server that IP, and your local DNS server goes out to that IP and says, “What’s the IP for www.google.com?” Google’s DNS servers give them an IP address and how long they can store that answer. When your local DNS server has an answer it likes, it gives that answer back to you.

Your web browser than opens a connection to that IP address on port 80 (the port for unencrypted web browsing traffic) and requests the main page (“GET /”, perhaps with a status number and header information about your browser and what kind of media you’re willing to accept and such, but the “GET /” is all that’s required. If you have telnet, you can actually do this manually and see what you get back). The www.google.com web server sends you back an unencrypted HTML file and closes the connection. If there are embedded images, your browser requests each of them in turn while it loads the page.

If you have a fast connection, this all typically happens in milliseconds. Well. Unless the page is full of ginormous images, hahaha.

If you instead request https://www.google.com, the DNS dance is the same but there are some additional moves on the actual file requests. Your browser gets the web site’s public key from the web server and uses that to request a symmetric key to encrypt the traffic, and then moves on to the file requests and deliveries. Oh, and this happens on 443 instead of 80, because that’s the default port. Unless your site has numbers on the end of the URL, like https://my401kprovider.com:8124 or something.

Aside: I previously mentioned a “man in the middle” attack; that’s where someone steps in between you and the web server and sends you a fake key and does the encrypt and decrypt dance between you, proxying the content. This is part of why you should pay attention to whether or not that padlock is green in your URL window, yes. There’s a whole system of who certifies whether the certificate is good; there are certificate authorities and your browser has some public keys for the authorities stored. (Some malware messes with those, too. Your antivirus might also be man-in-the-middling you to scan encrypted content for viruses. There’s some debate as to whether or not this is benign, and whether the advantages outweigh the risks–I think it depends on the user. You can tell both in your AV settings and by clicking on the padlock and asking for more info. Some AV companies also sell certs, so if you have Symantec or Comodo and the cert is from Symantec or Comodo that doesn’t prove anything, but if it says Avast, yeah, that’s their HTTPS scanning feature at work.)

The email client dance is similar; There’s the DNS dance of “who is mail.yourmailserver.com?” followed by really short requests sent by the software to authenticate you (your username and password) and get the content, and a similar encryption dance if you’re using encryption (which I recommend, yes). You can use telnet to send mail from the command line, which someone did in a previous embedded video.

Leave a Comment

Filed under written for nontechnical friends

Why Public Wifi is So Insecure

Here’s a short demo of sniffing.

They [TM] can possibly do this to you at that nice free public wifi spot you’re using.

They need to be on your network. This is geographically limited, by which I mean that if you’re in Kansas and visiting a server in Kansas, someone in England or Japan can’t see this stuff unless:

  • They’ve compromised something on your network,
  • They’ve compromised something between you and the site you’re visiting, or
  • They’ve compromised something near the host you’re talking to.

However, that shady person at the coffee shop’s free wifi could see this stuff just fine. These attacks typically occur somewhere near you.
They cannot see anything going over SSL, like web pages using https, unless they’re man-in-the-middling you, which is a whole different thing. Er. That’s more than I wanted to get into, but… it’s unlikely, unless the free wifi itself is messing with you.

Anyway. You can fire up Wireshark right now and possibly see your housemates’ passwords if they’re using insecure connections. (I once showed my ex his passwords over Wireshark.  Heh.)

What I’m getting at is:

  • Use SSL (https, secure connections on your email client, etc.) wherever you can.
  • Public wifi is super, super insecure. You probably want to use a VPN on public wifi.

Leave a Comment

Filed under written for nontechnical friends

Email Encryption

I’ve been trying to get people to encrypt their email, and send me encrypted email, since 1995 or so.  Here’s why:  email is cleartext.

What does “sent cleartext” mean?

Well, if your ISP is having you send your email out through port 25, you can do this.  (I’ve done it myself to test my own mail server at work.)

That’s basically what happens, automated, when you send your email out through a client. So unless your ISP has you send your mail out over SSL (they’ll probably give you another port to use, like 465 or 587), anyone sitting between you and your ISP’s outgoing mail server can potentially sniff that mail. (Like, that malicious person at the free wifi hotspot you’re using.)

So, you have an encrypted connection to your outgoing server, or you’re using https to access webmail. YAY! People sitting between you and that server can’t sniff it.

(Note: It’s even worse if you aren’t encrypting the connection to your incoming mail server in a mail client. Your username and password are sent cleartext over the wires.)

Don’t relax yet. 😉

What happens next is that your outgoing mail, or SMTP, server goes through the same process with your recipient’s SMTP server. This connection may or may not be encrypted and you have no way of knowing this unless your ISP refuses to talk to other servers that don’t encrypt. (Posteo.de says they do that.) Then the mail is stored on a server disk unencrypted until you or the recipient access it.

This is where encrypting the contents come in. If I send someone an encrypted email (*waves at [personal profile] cedara*), my outgoing mail server and their incoming mail server know that:

the mail came from my email address
the mail is addressed to their email address
there is a cleartext subject line.

The contents, however, look like:

Version: GnuPG v2


By the way, even though I’ve posted that to my blog, only the person who has the private key can read that.

This isn’t new technology. It’s actually old. The reason everyone isn’t using it is because it’s not set up by default, and because the people you email are probably not using it already and you can’t if they don’t. 🙁

If you’re a webmail user, there’s Mailvelope, which I haven’t used. For everyone else, Enigmail for Thunderbird is awesome. GPG4Win used to be kind of buggy but it’s gotten really good–I particularly like the “encrypt/decrypt clipboard” feature, which I’m using instead of Mailvelope for webmail.

Then, all you need is a recipient’s public key. (You always encrypt to the recipient’s public key.) The way it works, in short, is that each person has two keys: one public, one private. They give the public key to you and keep the private one a secret. Only the private key can decrypt something encrypted with its public key. So only one person can read what you’ve sent, and that’s the person who’s got the private key. (Here’s more.)

If you want to set it up and test it, I’ll email with you. My username @nym.hush.com will reach me. We can exchange public keys and go from there.

Leave a Comment

Filed under written for nontechnical friends

The Best Thing Windows Users Can Do To Prevent Malware

If you’re a Windows user, you probably want to do your daily web surfing as a non-administrative user. This is because any process you launch (on any operating system) runs with your account’s permissions. That means that if your browser runs into an exploit, that exploit basically runs as you.

My sister is a Mac user, so I know that when she goes to make an administrative change to her Mac, it asks her to confirm her username and password. I used to use Ubuntu on my primary system and can tell you that something similar happens there.

<tech>This makes sense, because they’re both based on unix and that credential prompt is based on sudo.</tech>

This isn’t how it works in Windows.

By default, the first user you set up in Windows is an Administrator, and subsequent users aren’t. (You can add or remove privileges later.) When you go to make an administrative change, since Vista Windows will ask you to click the much mocked “Are you sure?” button if you’re an administrator, or prompt for credentials if you aren’t. (Some people even turn the “Are you sure?” prompt off completely. Don’t!)

<tech>The “Are you sure?” button is technically called “UAC,” or User Access Control.</tech>

By using an unprivileged, non-administrative account, you force Windows to ask for account credentials. This limits the damage a browser exploit can do. It also means you have to remember two usernames and two passwords, but there you are. This also means that if the malicious process somehow manages to break out of UAC prison and bypass the “Are you sure?” prompt, it’s running as an account that’s not allowed to make those changes, anyway, and is out of luck.

Leave a Comment

Filed under written for nontechnical friends

Generate a Change Script to Move System Databases

I set this up to be modular and reusable, because I like reusing things.  This doesn’t actually move your system databases.  It just outputs a change script and a revert script.  You still have to run the generated script, then log on to your server and stop the services and move the files yourself.  In my example, I’m moving TempDB from C (ew) onto better storage.
declare @dbname sysname,
@oldpath varchar(255),
@newpath varchar(255)

set @dbname='tempdb'
set @oldpath='C:\Program Files\Microsoft SQL Server\MSSQLSERVER\MSSQL\DATA' --no trailing slash
set @newpath='H:\HappyStorage' --no trailing slash

/*  Generates your change script */
SELECT 'alter database ' + @dbname + ' modify file (name=' + name + ', filename = ''' + replace(physical_name,@oldpath,@newpath) + ''');'
FROM sys.master_files
WHERE database_id = DB_ID(@dbname);

/* Generates a revert script */
SELECT 'alter database ' + @dbname + ' modify file (name=' + name + ', filename = ''' + physical_name + ''');'
FROM sys.master_files
WHERE database_id = DB_ID(@dbname);

/* Tells you where the files are now */
SELECT name, physical_name
FROM sys.master_files
WHERE database_id = DB_ID(@dbname);

Sample output would  include, in the top window:

alter database tempdb modify file (name=tempdev, filename = 'H:\HappyStorage\Data\tempdb.mdf');
alter database tempdb modify file (name=templog, filename = 'H:\HappyStorage\Log\templog.ldf');

Copy and paste that into a new window, and run it if you’re sure it’s right.

You can, of course, use this for MSDB or any other database you want to move.

Leave a Comment

Filed under Uncategorized

De-duplicating files in batch

So, I have a drive full of files.  Sometimes, I get a new one–okay, sometimes I get over 1000 new ones–and want to keep the latest and pitch the duplicates without throwing away old files that don’t have a duplicate.  These files are of the form DATABASENAME_Attach_number_7777mmdd_hhmmss.sqb, but they could be anything, really.

(Why batch?  Because I had most of it lying around already and I’m lazy.)

set destination="i:\directory\with\old\files"
set source="f:\directory\with\new\files"
cd %destination%
mkdir old
mkdir new
move *.sqb old
robocopy %source% %destination%\new /mt:2 /mov
del early.txt
del files.txt
del names.txt
for %%F in ("old\*.sqb") do echo %%F >> early.txt
for /f "tokens=2 delims=\" %%A in (early.txt) do (
echo %%A >> files.txt
for /f "tokens=1-3 delims=_" %%A in (files.txt) do (
echo %%A_%%B_%%C >> names.txt
for /f %%F in (names.txt) do @if exist new\%%F* del old\%%F_FULL*
del early.txt
del files.txt
move names.txt dedupe
cd old
move * ..
cd ..\new
move * ..
cd ..
rd new
rd old

Leave a Comment

Filed under scripting

Pull the restore chain from MSDB

What files do I need to restore this database to a point in time?

You might know this off the top of your head, especially if you set up the backups or if you don’t have a lot of backups, but just in case you’re panicking, ask MSDB.  MSDB knows all… unless that data has been purged.  (Sorry.)

declare @dbname varchar(80),
@lastfull datetime,
@lastdiff datetime,
@fullback varchar(1024),
@diffback varchar(1024),
@RowsToProcess int,
@CurrentRow int,
@restorediff varchar(max),
@logback varchar(1024),
@restoreday datetime

set @dbname = ‘YourDB’
set @restoreday = ‘1/1/2016 8:05am’ –Make sure the time zone is your server’s time zone.

select @lastfull = max(backup_finish_date)
FROM master.sys.databases d
LEFT OUTER JOIN msdb.dbo.backupset b ON d.name = b.database_name AND b.type = ‘D’
WHERE d.database_id NOT IN (2, 3) and d.name=@dbname
and description like ‘Backup on%’
and backup_finish_date < @restoreday

SELECT    @fullback = m.physical_device_name
FROM         msdb.dbo.backupmediafamily AS m INNER JOIN
msdb.dbo.backupset AS b ON m.media_set_id = b.media_set_id
and b.type=’D’ and b.database_name=@dbname
AND b.backup_finish_date=@lastfull

print @fullback

SELECT @lastdiff = b.backup_finish_date FROM msdb.dbo.backupmediafamily AS m INNER JOIN msdb.dbo.backupset AS b ON m.media_set_id = b.media_set_id and b.type=’I’ and b.database_name=@dbname AND b.backup_finish_date>@lastfull AND b.backup_finish_date < @restoreday

set @lastdiff = @lastfull

select @diffback = m.physical_device_name FROM msdb.dbo.backupmediafamily AS m INNER JOIN msdb.dbo.backupset AS b ON m.media_set_id = b.media_set_id and b.type=’I’ and b.database_name=@dbname AND b.backup_finish_date=@lastdiff

print @diffback

CREATE TABLE #logs (RowID int not null primary key identity(1,1), logback nvarchar(255),finishdate datetime)
insert into #logs SELECT distinct m.physical_device_name, b.backup_finish_date FROM msdb.dbo.backupmediafamily AS m INNER JOIN msdb.dbo.backupset AS b ON m.media_set_id = b.media_set_id and b.type=’L’ and b.database_name=@dbname AND b.backup_finish_date>@lastdiff AND b.backup_finish_date<@restoreday order by b.backup_finish_date

SELECT @RowsToProcess=COUNT(logback) from #logs

SET @CurrentRow=0
WHILE @CurrentRow<@RowsToProcess
SET @CurrentRow=@CurrentRow+1
FROM #logs
WHERE RowID=@CurrentRow
print @logback

drop table #logs

Leave a Comment

Filed under Uncategorized

Windows Internal Database/SQL Express backups and maintenance

Google informs me that my post on Windows Internal Database Maintenance is popular.  It’s not what I’m currently using, however.

I’ve switched over to Ola Hallengren’s scripts.  If you download and run the installer script, it creates a series of agent jobs, which, well.  Neither Windows Internal Database nor SQL Express has SQL Server Agent.  However, the jobs run as cmdexec scripts that look like

sqlcmd -E -S $(ESCAPE_SQUOTE(SRVR)) -d master -Q "EXECUTE [dbo].[DatabaseBackup] @Databases = 'USER_DATABASES', @Directory = N'f:\sql_backups', @BackupType = 'FULL', @Verify = 'Y', @CleanupTime = 672, @CheckSum = 'Y', @LogToTable = 'Y'" -b

It’s easy to run these through Task Scheduler.  You’ll want to replace “$(ESCAPE_SQUOTE(SRVR))” with your actual server.

(The connection string for the version of Windows Internal Database based on 2008 is \\.\pipe\MSSQL$MICROSOFT##SSEE\sql\query.  The connection string for the version of Windows Internal Database based on 2012 is \\.\pipe\MICROSOFT##WID\tsql\query.  SQLExpress on your current host is .\SQLExpress by default.)

Continue reading

Leave a Comment

Filed under scripting, sql

Nagios Checks for Dell OpenManage Disk Health Written in PowerShell

I wrote these in PowerShell to run with nrpe/nsclient.  They query Dell OpenManage’s command line and return a Nagios-readable result.

(There are plugins available on Nagios Exchange, but they all seemed to be… more than I wanted.  I just wanted to know, “Is my RAID okay?”  Because whenever I’ve had an Event, I like to monitor for that event happening again.)

This checks the Physical Disks in the array.  If one or more disks reports anything other than OK, it alerts:

$status = 0; omreport storage pdisk controller=0 | Where-Object {$_ -match "^status"} | %{if($_ -notlike "*OK*"){$status=2}}

If ($status -eq 0) {
Write-Host "OK:  Physical Disks report OK"
} else {
Write-Host "CRITICAL:  Check OpenManage"
exit $status

You might have to edit the scripts to check Virtual Disk health.  It could probably be made more elegant, but it suits my purposes.

This script checks the health of my C drive (vdisk 0):

omreport storage vdisk controller=0 vdisk=0 | ?{$_ -match "^status"} | %{$status=0}{if($_ -notlike "*OK*"){$status=2}}

If ($status -eq 0) {
Write-Host "OK:  Virtual Disk (OS) reports OK"
} else {
Write-Host "CRITICAL:  Check OpenManage"
exit $status

This script checks the health of my data drive (E, vdisk 1):

omreport storage vdisk controller=0 vdisk=1 | ?{$_ -match "^status"} | %{$status=0}{if($_ -notlike "*OK*"){$status=2}}

If ($status -eq 0) {
Write-Host "OK:  Virtual Disk (data) reports OK"
} else {
Write-Host "CRITICAL:  Check OpenManage"
exit $status

Running these involves adding lines like this to the external script section of nsclient.ini or equivalent:

check_physicaldisk = cmd /c echo scripts\\pdiskcheck.ps1; exit($lastexitcode) | powershell.exe -command -

check_virtualdisk = cmd /c echo scripts\\vdiskcheck.ps1; exit($lastexitcode) | powershell.exe -command -

And lines like this to command.cfg in Nagios:

define command {
command_name    check_physicaldisk
command_line    /usr/lib/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c check_physicaldisk
define command {
command_name    check_CRaid
command_line    /usr/lib/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c check_virtualdisk

Leave a Comment

Filed under Uncategorized

Katherine’s Excellent Log-Shipping Adventure

Or, Log-Shipping over 1200 databases automagically.


Continue reading

Leave a Comment

Filed under monitoring, powershell, sql