Category Archives: PowerShell

Downloading Files with PowerShell and Invoke-WebRequest

Last weekend I was at the Atlanta Code Camp, giving a presentation on PowerShell for Developers. One of the attendees emailed me, asking for a good example of being able to download multiple files from a website.

To do so, we’ll use the Invoke-WebRequest cmdlet. But first, we’ll setup a directory to hold our output. For all my demos I have a root folder, C:\PowerShell. For this article I’ve created a subfolder called Invoke-WebRequest-Demo. The first thing my code does is sets a variable to point to this folder, then changes the current location to it.

$dir = 'C:\PowerShell\Invoke-WebRequest-Demo'
Set-Location $dir

Next, we’ll need a variable to point to the root URL of the website we wish to download from. This will be everything except for the actual file name.  For this demo I will use files from the No Agenda Show podcast. Not only is it a cool podcast, but they have an unrestricted license for their material meaning I can freely reuse it for demo purposes. Each episode of the podcast has it’s own image, so we’ll download images from a few recent episodes.

After I place the base URL into a variable, I create an array, each item in the array having the name of a file to download.

$baseUrl = 'http://adam.curry.com/enc/'
$files = '1537127586.656_na-1069-art-feed.png',
          '1536868338.385_na-1068-art-feed.png',
          '1536264075.484_na-1066-art-feed.png'

Now that everything is setup, we use a simple foreach loop to iterate over the array and download each file via Invoke-WebRequest.

foreach ($file in $files)
{
   Write-Host "Downloading $file"
   $dlUrl = "$($baseUrl)$file"
   $dlPath = "$($dir)$file"
   Invoke-WebRequest $dlUrl -OutFile $dlPath
}

And that’s all there is to it. If you want to learn more about downloading files via the web, this code was extracted from my Testing PowerShell with Pester course on Pluralsight. In it I test a module which gets the RSS feed, then downloads images and audio files for the No Agenda show. Just go to the About Me link at the top and you’ll find a complete list of all my Pluralsight courses.

Speaking of Pester, you might also appreciate the introduction to Pester series of articles I’m currently authoring for Red Gate’s Simple Talk website. You can find a link via the same About Me page, or just jump directly there by going to http://arcanecode.red to see my articles.

Advertisements

Get Notified When Your Home Router IP Address Changes–With PowerShell!

Readers of my blog over may recall I’m an amateur (ham) radio operator (N4IXT that’s me!). One of my ham radio buddies has a cool radio that can be remotely controlled over the internet. With their software he just enters the IP address of his home (his home router that is, after we setup some port forwarding), and he can connect to and operate his radio remotely, make contacts around the world and more.

The tricky part of all this came in the sentence enters the IP address of his home…. His home router tends to change its address on the whim of his service provider. If the router reboots, it will definitely have a new address when it comes up. You may be thinking “well just get a static IP address”. Unfortunately, his internet service provider can’t give him this, something about him using a bonded pair of 10 mb lines to get 20 mb prevents it.

So how could he get notified when his IP address changes? Well I was fairly certain I could solve his problem with some simple PowerShell scripting!

I’ve now published the solution on my GitHub site:

https://github.com/arcanecode/PowerShell/tree/master/IPAddressMailer

Rather than going more in-depth here, the ReadMe has the full details describing the scripts that were created. In addition the code is heavily commented.

In addition to solving this particular problem, the code could be used as a template for:

  • Getting the IP Address of your home router
  • Sending emails via GMail
  • Setting up tasks in the Windows Task Scheduler

Hopefully other people with similar issues will find this little project as useful as my ham radio buddy has!

Azure PowerShell PlaybooK: Azure SQL–Now on Pluralsight!

My latest course is now available on Pluralsight! It’s the Azure PowerShell Playbook: Azure SQL. If you aren’t familiar with Pluralsight’s Playbook format, they are fast past courses that are almost 100% demo driven. They are meant to be consumed quickly, my course is just one hour and four minutes long. Great lunchtime viewing!

This course shows you how to use PowerShell to manage and migrate your on premises database up to Azure SQL. In brief, you’ll learn how to:

  • Create resource groups
  • Create and work with Storage Accounts
  • Create a SQL Server in Azure SQL
  • Package up your local database into a bacpac file
  • Import your bacpac file into a new Azure SQL database
  • Execute commands against your new Azure SQL database
  • Cleanup! I even show how to remove everything you’ve created, individually or as a whole

And all of this with PowerShell!

Additionally, I’ve included functions for just about everything listed, so (assuming your subscription gives you access to the samples) you’ll have a great starting point for your own library of cmdlets. (All the code for the functions appears on screen, so if you have to you could always pause and type it in.)

You can find my new course at:

https://www.pluralsight.com/courses/azure-powershell-sql-playbook

I also wrote an article for RedGate’s SimpleTalk website that aligns well with this course. I dive deeper into the restartability aspect of the way the functions were coded, something I couldn’t get deep into with the video course due to time constraints.

https://www.red-gate.com/simple-talk/sysadmin/powershell/powershell-functions-reusability-restartability-azure/

What’s that? Yes you in the back row, you say you don’t have a Pluralsight subscription? Well no worries, just email me, free@arcanetc.com and I’ll be glad to send you a code that will be good for 30 days at Pluralsight. During that time you can watch my courses, indeed you can watch any course at Pluralsight.

Creating Azure Resource Groups Safely With PowerShell

A few posts back I mentioned that I had become a SimpleTalk author, and that my first post for them was PowerShell Functions For Reusability and Restartability in Azure (read it at http://bit.ly/acred01)

I’ve created a companion video for it, which focuses on creating Resource Groups. You can see it below, or on YouTube at http://bit.ly/acredyt01.

As I mention in the video, you can find the code samples on my github site, at http://bit.ly/acredgit01

I’m Now a SimpleTalk Author!

RedGateSimpleTalkLogoVerticalAdding to my other activities, I’m now writing for SimpleTalk, RedGate’s community hub. My first article just went live.

PowerShell Functions for Reusability and Restartability in Azure is the title, in it I describe how to implement the concepts of reusability and restartability in PowerShell. The functions were written against the Azure platform, however the concepts are valid for any PowerShell implementation. You can read the full article at:

https://www.red-gate.com/simple-talk/sysadmin/powershell/powershell-functions-reusability-restartability-azure/

If you want to keep up with all my articles, I’ve setup a special URL which will jump you to my author page at SimpleTalk:

http://arcanecode.red

It’s small now, but expect it to grow quickly. Happy reading!

Eric Ligman’s FREE Microsoft eBook Giveaway–Revising the download script

Every year, Eric Ligman, director of Sales Excellence for Microsoft, creates a blogpost in which he gives away tons of FREE Microsoft eBooks. This year has 361 in the list.

You name it, it’s in the list. SQL Server, Azure, PowerShell, .NET, BizTalk, SharePoint, Windows Server, and more. You can find Eric’s post at:

https://blogs.msdn.microsoft.com/mssmallbiz/2017/07/11/largest-free-microsoft-ebook-giveaway-im-giving-away-millions-of-free-microsoft-ebooks-again-including-windows-10-office-365-office-2016-power-bi-azure-windows-8-1-office-2013-sharepo/#comments

While there are individual links to each file, what if you want every one of them? He explains on the post why he doesn’t provide a big zip file. He does, however, provide a PowerShell script (attributed to David Crosby) that will do the job.

However, I found some issues with the script. Not that it didn’t work, it did, but there were several things I felt could be done to improve it.

First, there was no progress message issued during the download. As a user, I had no idea which file I was on, so had no concept of how much longer it would take. Thus, I’ve added a little progress message.

I then thought “Hmm, what if my downloads were interrupted, I don’t want to have to start all over”. So, I added some code that sees if the file we’re downloading already exists. This way it won’t re-download a file it already has.

But then another problem arose. What if it had partially downloaded a file? Just checking the file names wouldn’t catch that. So I added further code to compare the file size at the source with the file size on disk. If different, then it will re-download.

So far so good, now it will skip the file only if the file name is already on the local disk, and the file sizes match.

I now encountered my next concern. Crappy internet. I live out in the country, and while I love my privacy and rural living, my internet sucks. It is prone to go down or drop packets. If it had issues during a  download I didn’t want it to crash, but instead go onto the next file.

Thus I added a try/catch error handler, which displays an error message and continues on.

At this point I thought I was done. Just I was about to call it finished though, a typical afternoon Alabama thunderstorm came up. Kaboom! House rattled and power blinked.

This presented my final concern, what if the power went out? I’d want to know where it got to with the downloads. So I added some further code such that when the downloading starts it creates a new log file and appends each message to it.

I realize some of you have superfast gigabit internet and will be able to download these almost instantly. (I hate you by the way. #jealous). Therefore I made logging optional, so it wouldn’t create an extra file if you didn’t want it. Just set the $log variable to $false, and it will skip logging.

So there you go, a revised download script that will handle stopping and restarting the script gracefully, will look for errors, and adds logging so you can track progress.

You’ll find the revised script on my GitHub site, in the PowerShell folder:

https://github.com/arcanecode/PowerShell

Just look for the file “Eric Ligmans Microsoft eBook Giveaway Revised Download Script.ps1

There’s also a readme style file by the same name, which echoes this blog post.

What happened to Save-AzureRmProfile?

I’ve been working a lot in the Azure PowerShell area of late. One thing I wanted to be able to do is have my scripts login automatically to Azure. In many examples the cmdlet Save-AzureRmProfile was used to save your Azure credentials, then later you could use Import-AzureRmProfile to import them.

But, when I attempted to run Save-AzureRmProfile I got the error ‘Save-AzureRmProfile is not recognized as the name of a cmdlet, function, script file, or operable program’.  Huh? I checked the docs, and it does include a listing for Save-AzureRmProfile.

https://docs.microsoft.com/en-us/powershell/module/azurerm.profile/save-azurermprofile?view=azurermps-3.8.0

This is a case of the PowerShell AzureRM module getting ahead of the docs. After beating my head against the wall, I found the cmdlets had been replaced with the new noun of AzureRmContext.

To use them, first login to Azure manually. Then, use the new Save-AzureRmContext to save your information to a file.


# Setup – First login manually per previous section
Add-AzureRmAccount

# Now save your context locally (Force will overwrite if there)
$path = "C:\Azure\PS\ProfileContext.ctx’
Save-AzureRmContext -Path $path -Force

Once that’s done, from then on you can use the Import-AzureRmContext to automate the login.


# Once the above two steps are done, you can simply import
$path = C:\Azure\PS\ProfileContext.ctx’
Import-AzureRmContext -Path $path

Be warned, this does present a security issue. If someone were to steal your context file, they could then login as you. You need to be sure your context file is stored in a safe location no one can get to.