Questions & Help>Questions

Strange behavior with SHA vs phash2

#crypto#hash
darkmarmot

darkmarmot

Strange behavior with SHA vs phash2

We have some code that creates unique hashes on documents to handle duplicate checks.

It wasn’t working on nodes that rebooted (letting dupes through) and as a last resort I changed from SHA to phash2 and it works perfectly now. This seems really strange to me. Is there some kind of timestamp seed or something in the crypto SHA I’m not aware of?

Here’s the code change that fixed our issues:

hash =
  :erlang.phash2(doc, 1_000_000)

Here’s the original that is not working across restarts:

hash =
  :crypto.hash(:sha, :erlang.term_to_binary(doc)) |> Base.encode64(case: :lower)

Maybe term_to_binary or encode64… but… it just surprised me a lot…

2 comments
#hash
5 1346 2

Most Liked

cmkarlsson

cmkarlsson

Sorry for a few follow-up questions. I am not even sure I can help here.

  • Can you consistently reproduce this, the same document always produce different results after reboot of a node?
  • What is doc? Is it a binary string or an erlang term? Something else?
  • Is the initial hash and hash done on rebooted node done on the same nodes?
  • If on different nodes are they running on different server architecture?
  • Are you able to try the different parts of the sha hash individually to see which one is not working.

My bet is that it is term_to_binary. I am not sure it is meant to always produce the same result, only that it can be decoded with binary_to_term.

phash2 on the other hand is designed to always produce consistent result when run on different version of erlang and machine architectures.

2
Post #2
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

The collision chance for phash2 is gonna be a lot higher than a sha though, so I’d definitely recommend finding a way to use a sha.

2
Post #3

Where Next?

View thread on forum (has 2 responses!)
crypto
hash
Home Questions & Help>Questions
#crypto#hash
5 1348 2
Last post

Popular in Questions Top

Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32139 16
New
lessless
Questions & Help>Questions

How are you dealing with CSV files that uses CR line breaks?
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
#how-to-question
10 18353 17
New
jerry
Questions & Help>Questions

Ecto query using like/ilike in query
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
#ecto
17 16504 10
New
Werner
Questions & Help>Questions

Mix local.hex warning - ‘Authenticity is not established by certificate path validation’
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
#troubleshooting
47 14042 14
New
myronmarston
Questions & Help>Questions

Typespecs - best way to spec keyword lists?
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
#dialyzer
52 14258 3
New
ashish173
Questions & Help>Questions

Difference in between :utc_datetime and :naive_datetime in Ecto
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
#ecto#timestamps
79 27907 15
New
gonzofish
Questions & Help>Questions

Ecto: how to join three tables?
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
/phoenix#ecto
21 13889 7
New
shahryarjb
Questions & Help>Questions

How to convert map to string (separated with ,)
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
#maps#strings
15 15491 2
New
LegitStack
Questions & Help>Questions

Websocket server in Elixir or phoenix
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
/phoenix#websockets
20 11315 8
New
chewm
Questions & Help>Questions

how configure visual studio code for elixir intellisense
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
#code-editors#vscode#vscode-elixir
7 14611 11
New

Other popular topics Top

Qqwy
Blogs & Podcasts>Blog Posts

Elixir Blog Posts
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
hexdocs.pm
#blog-posts#wiki#stickies#official-blog-posts-thread
3268 119930 1237
New
chrismccord
News>Phoenix News

Phoenix 1.4.0 released!
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
/phoenix#phoenix-release
688 30048 115
New
peerreynders
Learning Resources>Chat/questions

Elixir Book & Learning Resources Sales / Discounts Thread
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
#discounts
326 29600 154
New
srinivasu
Questions & Help>Questions

Best Practises for Error handling elixir?
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
16 16061 9
New
sergio
Learning Resources>Guides/Tuts/Tips/Info

Phoenix 1.6.0 LiveView + esbuild + Tailwind JIT + AlpineJS - A brief tutorial
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
/phoenix#liveview
137 17627 35
New
malloryerik
Learning Resources>Guides/Tuts/Tips/Info

How to use 'phoenix-heex' templates in VSCode
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
/phoenix#vscode#heex
118 16932 34
New
lk-geimfari
Questions & Help>Questions

The most correct way to read a JSON file
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
23 13679 2
New
vertexbuffer
Questions & Help>Questions

Deleting item from a list
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
7 24097 4
New
chrisalley
Questions & Help>Questions

How to describe many contexts in ExUnit without a hierarchy
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
#testing#exunit
66 16773 4
New
AstonJ
Chat & Discussions>Wikis

Phoenix LiveView Info
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
/phoenix#wiki#stickies#liveview
220 25307 74
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Manning

Develop your skills with books, videos, and courses.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement