Thursday, July 6, 2017

postgresql corruption: ERROR: tuple concurrently update (duplicate key in primary key column)


I am not exactly sure how to classify this type of issue. It seems like a pg_toast out of sync issue or typle concurrently update error. Either way, the table have an exact records where PK has been defined.

The workaround is to delete one of the duplicate record from the table.

The only way to delete the record is with using CTID instead of any ID defined in the column with PK as the record itself has 2 identical PK key, so it would not worked. It will wiped out both records and one is needed.


Typical errors
delete from vpx_vm where id=2398;
ERROR:  tuple concurrently updated


There are times, select on the table may be fine but the example id could return 2 of the identical records. The workaround is to delete one of them so it would not have duplicates. The trick is which on to delete ?

select ctid, id from vpx_vm where id = 1433; -- this may return 2 records with 2 different CTIDs. One of the 2 records will result of unexpected chunk number.







Example: 
select * from vpx_vm where ctid = (8,2);   fine
select * from vpx_vm where ctid = (69,2); - error out with pg_toast missing chunk 

delete the bad ctid
delete from vpx_vm where ctid = (69,2),    track that vm from vpx_host too.


Select each of the rows with their corresponding CTID and one of them will fail. Delete the one that is failed and leave the one return output alone.
select  * from vpx_vm where ctid = xxxx


Technique that have been used to address this issue and I found out that it may not be that efficient as there is a chance that the good rescord to be deleted and the corrupt record left in the table.

delete from vpx_vm where ctid=(select ctid from vpx_vm where id=2398 limit 1);  

The above issue is fairly common, typically found right after disk filled up and database crashed.

Saturday, January 7, 2017

postgresql: corruption: ERROR: unexpected chunk number 0 (expected 1) for toast value XXXXXXXXX in pg_toast_XXXX


This issue is very common but luckily, they can be resolved majority of the time (it was once it did not worked for me). The corruption is a bit difficult to description and I am using a real scenario here.

The vcenter crashed and in the logs, the error reported was.


2017-01-05T15:51:06.640Z [7FC61081D700 error 'Default' opID=SWI-3a1cc46] [VdbStatement] SQLError was thrown: "ODBC error: (XX000) - ERROR: unexpected chunk number 0 (ex
pected 1) for toast value 25270341 in pg_toast_16875;
--> Error while executing the query" is returned when executing SQL statement "SELECT IS_CONSOLIDATE_NEEDED, DAS_PROTECTED_ACTUAL, DAS_PROTECTED_DESIRED, MANAGED_BY_TYP
E, MANAGED_BY_EXT_KEY, STORAGE_SPACE_UPDATED_TIME, AGGR_UNSHARED_STORAGE_SPACE, AGGR_UNCOMMITED_STORAGE_SPACE, AGGR_COMMITED_STORAGE_SPACE, RECORD_REPLAY_STATE, FAULT_T
OLERANCE_STATE, PENDING_ANNOT_SET_FLG, PENDING_NAME, ONLINE_STANDBY, MKS_CONNECTIONS, SCREEN_WIDTH, SCREEN_HEIGHT, TOOLS_MOUNTED, MEMORY_OVERHEAD, SUSPEND_INTERVAL, BOO
T_TIME, SUSPEND_TIME, UUID_INSTANCE, UUID_BIOS, NUM_DISK, NUM_NIC, NUM_VCPU, MEMORY_RES"


When trying to delete the offending entry from the vpx_vm using the id, it throws a different errors.

ERROR: tuple concurrently updated

Backing up the table will throw the same problem happens.

ERROR: unexpected chunk number 0 (expected 1) for toast value 25270341 in pg_toast_16875;


Steps to resolve the issue for VCSA.


1: Back up the database first.

2: Stop vpxd & vdcs service:
     service vmware-vpxd stop
     service vmware-vdcs stop

3: Connect to VCDB with:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

4: Execute below query:
   select pg_toast_16875::regclass;

   Above query should give you output something like:
   regclass
   --------
   VPX_VM

5: Now re-index the table with below queries:
   REINDEX table pg_toast.pg_toast_16875;
   REINDEX table VPX_VM; (Table name should be whatever is returned from step#4)

6: Do vacuum and analyze for vpx_vm table with following query:
    VACUUM ANALYZE VPX_VM; (Table name should be whatever is returned from step#4)

7: \q

8: Start vpxd & vdcs service:
   service vmware-vpxd start
   service vmware-vdcs start

NOTE: The steps can lead to some data loss from table.

The key is to locate the pg_toast then reindex the pg_toast and its corresponding table.

Friday, November 18, 2016

postgresql: pg_relation_size glitches?

How did pg_relation_size locate something does not exist in the db?

VCDB=# select current_database(), version();
 current_database |                                                                     version

------------------+---------------------------------------------------------------------------------------------------------------
-----------------------------------
 VCDB             | PostgreSQL 9.4.9 (VMware Postgres 9.4.9.1-4536683 release) on x86_64-unknown-linux-gnu, compiled by x86_64-vmk
-linux-gnu-gcc (GCC) 4.8.4, 64-bit
(1 row)

VCDB=# select pg_relation_size(table_name) as Size  FROM information_schema.tables WHERE table_schema NOT IN ('information_schema', 'pg_catalog') ORDER BY size DESC LIMIT 10;
ERROR:  relation "vci_version" does not exist

VCDB=# \d vci_version;
Did not find any relation named "vci_version".
VCDB=#

VCDB=#  select table_name FROM information_schema.tables WHERE table_schema NOT IN ('information_schema', 'pg_catalog');
           table_name
--------------------------------
 cis_kv_providers
 cis_kv_keyvalue
 vpx_alarm_expression
 vpx_alarm_expr_comp
 vpx_alarm
...

SELECT tablename,  pg_size_pretty(pg_relation_size(schemaname || '.' || tablename))
FROM pg_tables WHERE schemaname = 'vc';

           tablename            | pg_size_pretty
--------------------------------+----------------
 cis_kv_providers               | 8192 bytes
 cis_kv_keyvalue                | 8192 bytes
 vpx_alarm_expression           | 112 kB
 vpx_alarm_expr_comp            | 48 kB
 vpx_alarm                      | 104 kB
 vpx_alarm_runtime              | 8192 bytes


Tuesday, October 18, 2016

postgresql: Confusing piece of search_path in vcdb.


For some reason, every time I need to look at the tables vc owned, even login in with "vc", I have to perform the following to look for objects in the db. This isn't causing much problem but accessing with Python pyscopg2, it will not find the tables.


set search_path to public, vpx;



vcdb=> select usename, useconfig from pg_user
vcdb-> ;
 usename  |      useconfig
----------+----------------------
 postgres |
 vc       | {search_path=public}
(2 rows)


No matter how I changed the search_path in the postgresql.conf. A fresh login just wouldn't let me find my own tables. Finally, I decided to just let database search_path.


vcdb=> alter database vcdb set search_path to public, vpx, vc, postgres;
ALTER DATABASE
vcdb=> \d
No relations found.
vcdb=> \c vcdb
psql (9.4.9, server 9.2.9)
You are now connected to database "vcdb" as user "vc".
vcdb=> \d
                     List of relations
 Schema |              Name              |   Type   | Owner
--------+--------------------------------+----------+-------
 vpx    | vpx_access                     | table    | vc
 vpx    | vpx_access_seq                 | sequence | vc
 vpx    | vpx_alarm                      | table    | vc
 vpx    | vpx_alarm_action               | table    | vc
 vpx    | vpx_alarm_disabled_actions     | table    | vc
 vpx    | vpx_alarm_expr_comp            | table    | vc
 vpx    | vpx_alarm_expression           | table    | vc
 vpx    | vpx_alarm_repeat_action        | table    | vc
 vpx    | vpx_alarm_runtime              | table    | vc
 vpx    | vpx_alarm_seq                  | sequence | vc
 vpx    | vpx_alarm_state                | table    | vc
 vpx    | vpx_ansfile_seq                | sequence | vc
 vpx    | vpx_array_seq                  | sequence | vc
 vpx    | vpx_binary_data                | table    | vc
 vpx    | vpx_binary_data_seq            | sequence | vc
 vpx    | vpx_bulletin_operation         | table    | vc
 vpx    | vpx_change_tag                 | table    | vc
 vpx    | vpx_change_tag_seq             | sequence | vc
 vpx    | vpx_compliance_status          | table    | vc
 vpx    | vpx_compliance_status_seq      | sequence | vc
 vpx    | vpx_compute_res_failover_host  | table    | vc
 vpx    | vpx_compute_res_user_hb_ds     | table    | vc
 vpx    | vpx_compute_resource           | table    | vc
 vpx    | vpx_compute_resource_das_vm    | table    | vc
 vpx    | vpx_compute_resource_dpm_host  | table    | vc
vcdb=>


postgres=# SELECT r.rolname, d.datname, rs.setconfig
postgres-# FROM   pg_db_role_setting rs
postgres-# LEFT   JOIN pg_roles      r ON r.oid = rs.setrole
postgres-# LEFT   JOIN pg_database   d ON d.oid = rs.setdatabase;
 rolname | datname |                 setconfig
---------+---------+-------------------------------------------
 vc      |         | {"search_path=public, vpx"}
         | vcdb    | {"search_path=public, vpx, vc, postgres"}
(2 rows)




Voila, Everybody can access now!

Monday, October 17, 2016

postgresql: one of the few ways to start postgresql db under different port.


nohup /usr/pgsql-9.3/bin/postgres -p 5433 -D /var/lib/pgsql/9.3/data &

OR

--Needs double code between the port. Otherwise, it will complain. Kept forgetting this after several weeks. The silly thing is, it should have complained about the -p instead of the port number.


-bash-4.1$ /usr/pgsql-9.3/bin/pg_ctl "-p 5433" -D /var/lib/pgsql/9.3/data start
sh:  5433: command not found
server starting




/usr/pgsql-9.3/bin/pg_ctl -o "-p 5433" -D /var/lib/pgsql/9.3/data start



-bash-4.1$ ps -ef|egrep '5432|5433'|grep -v grep
postgres  2040     1  0 Apr27 ?        00:04:44 /usr/pgsql-9.2/bin/postmaster -p 5432 -D /var/lib/pgsql/9.2/data
postgres 14362     1  0 16:48 pts/3    00:00:00 /usr/pgsql-9.3/bin/postgres -D /var/lib/pgsql/9.3/data -p 5433


Wednesday, July 6, 2016

postgresql: Backup postgresql database

Postgresql has 2 ways of back up.

The first is pg_dump

A simpler form of pg_dump which typically using superuser or the database owner.

pg_dump vcdb > vbdb_db.bak

Backing database for VMware vPostgresql database.

localhost:~ # /opt/vmware/vpostgres/9.3/bin/pg_dump -U postgres -d VCDB -Fc -b -v -f vcdb.backup.compressed




A second method of back will be backup in form of ASCII. 

/opt/vmware/vpostgres/9.3/bin/psql -U postgres -d VCDB > vcdb.bk


Tuesday, May 10, 2016

postgresql: drop function with return value without knowing it.

Unlike Oracle, Postgresql do not distinct between procedure but considering procedure as function with out return value. Technically, they are the same just procedure without return value  but would be nice to differentiate them. The inconvenient comes when Procedure and Function are considered as the same object type in Postgresql which is function. Dropping a function without parameter would be the way way but what about dropping function with parameter? It is not as easy as Oracle in this matter. Came up with this script and I was able to drop it.



-- Check if the function in the database

vcdb=# \df remove_cafe_resource;
                                      List of functions
 Schema |         Name         | Result data type |       Argument data types       |  Type
--------+----------------------+------------------+---------------------------------+--------
 public | remove_cafe_resource | void             | resource_name character varying | normal
(1 row)


-- Attempt to drop the function without parameter it should fail but telling me that it does not exist instead.

vcdb=# drop function remove_cafe_resource();
ERROR:  function remove_cafe_resource() does not exist

-- Script to drop it. Copy and paste it to drop the function.
select 'drop function ' || oid::regproc  || '(' || pg_get_function_identity_arguments(oid) || ');' FROM   pg_proc WHERE  proname = 'remove_cafe_resource' AND pg_function_is_visible(oid);

                               ?column?
----------------------------------------------------------------------
 drop function remove_cafe_resource(resource_name character varying);
(1 row)


-- Dropped
drop function remove_cafe_resource(resource_name character varying);
DROP FUNCTION

vcdb=#  \df remove_cafe_resource;
                       List of functions
 Schema | Name | Result data type | Argument data types | Type
--------+------+------------------+---------------------+------

(0 rows)