Include URL percent-encoding?

[KorbisInc]

For anyone finding needing this

Read #219 (reply in thread) 🙂

Original text:

---

Is it too out-of-scope to wish for encoding and decoding URL percent-encoding? A quick search shows there are dozens of code-golfed one liners in the wild, but nothing as simple as echo $URL | trurl --%-decode... (It's nice that % isn't a special character in bash...)

A problematic example is trying to use curl on something like curl https%3A%2F%2Fsecure.gravatar.com%2F - curl will report a bad hostname. It'd be great if anyone could do trurl --%-decode https%3A%2F%2Fsecure.gravatar.com%2F | curl

[KorbisInc]

Seeing: #200 (comment) "trurl only outputs valid URLs" - I believe then only --%-decode should be supported.

Edit: nevermind, percent-encoded URLs are valid! curl/curl#7830

[emanuele6]

@LeeKorbis

A problematic example is trying to use curl on something like curl https%3A%2F%2Fsecure.gravatar.com%2F - curl will report a bad hostname. It'd be great if anyone could do trurl --%-decode https%3A%2F%2Fsecure.gravatar.com%2F | curl

Ok, that example helps! Now I understand more what you want. Your $URL is not a URL itself, but it is a string that when percent decoded is a URL.

If that URL comes from a query in the URL, you can just use something like, that is fine:

$ trurl -g '{query:redir}' 'http://example.org/?redir=https%3A%2F%2Fsecure.gravatar.com%2F'
https://secure.gravatar.com/

If it is just a string, and not a part of a URL, and you must use trurl because you cannot use any other tool to urldecode your string so you have no other option, you could use an hack like:

$ trurl -s 'fragment:=https%3A%2F%2Fsecure.gravatar.com%2F' -g '{fragment}' localhost
https://secure.gravatar.com/

(note {:fragment} and {fragment} have accidentally been swapped in trurl 0.7, so for trurl 0.7 specifically that should be {:fragment}, but that is a bug)

[KorbisInc]

Haha, your first example is the exact situation I'm in. To be even more specific, I was manually collecting image examples off of Yandex and put the URLs into a text file, and found that img_url held the original URL to the images. A couple of editor kung-fu moves later I had just the percent-encoded URLs but saw that Chromium, then wget, then finally curl couldn't download from them.

-g '{query:img_url}' will resolve this problem quite easily 😎

Thank you again and stay awesome

[emanuele6]

And note that trurl can also get URLs from stdin, so if you have a file that contains those URLs, and you want to extract {query:img_url} for each one, you can just use:

trurl -f - -g '{query:img_url}' < baseurl > imgurls

Or if you want to edit the file inplace, you can use :%!trurl -f - -g {query:img_url} in vim. B)

[Indyandie]

Would it be a bad idea to have a way to encode a URL or parts of a URL, like query params?

[emanuele6]

@Indyandie I don't really understand what you mean, it sounds like what trurl already does normally. Can you open a new discussion and clarify?